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additional teachers in schools participating in a large-scale urban effort. 
Overall the reliability and consistency of the assessment ratings were good. 
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USING TEACHERS' ASSIGNMENTS AS AN INDICATOR 
OF CLASSROOM PRACTICE 



Lindsay Clare 

CRESST/University of California, Los Angeles 
Abstract 

This report describes research developing indicators of classroom practice for monitoring 
the influence of school reform initiatives on students' learning environments, and 
supporting the improvement of instructional practice. The work reported here entailed 
collecting assignments and student work and observing classrooms in schools 
participating in a large-scale urban effort. Overall the reliability and consistency of the 
assignment ratings were good. Results also indicate that the quality of the assignments 
was statistically significantly associated with the quality of observed instruction and 
student work. It appears that our method shows promise for use in large-scale evaluation 
settings and identifies important dimensions of practice that could support teacher self- 
evaluation and reflection. 



Since the 1980s there has been a proliferation of reforms intended to improve 
the quality of student achievement, especially for those students who are at the 
greatest risk for academic problems. In spite of the adoption of many reform 
programs in urban schools, however, there has been little change in the academic 
welfare of poor and minority students, who continue to perform well below state 
and national norms on standardized tests of achievement. Quality of teaching is the 
single most important school factor in determining student success (Tharp & 
Gallimore, 1988). According to the National Commission on Teaching and America's 
Future (1996), however, many teachers do not have the knowledge and skills needed 
to significantly improve the quality of the learning environments they create for 
students. It appears that reform efforts are not "working," in part because teachers 
are not able to achieve the goals for learning and instruction set out in many 
programs (see, for example, Cohen & Ball, 1994). 

Despite the fact that the effectiveness of school reform efforts for raising 
student achievement ultimately depends on the quality of instruction, we do not yet 
have effective and efficient ways to describe classroom practice and directly monitor 
the influence of reform efforts on students' learning environments. Teacher surveys 
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frequently have been used to indirectly assess the quality of students' learning 
environments, though this method has limitations as far as yielding accurate 
information about the quality of teacher-student interactions and engagement with 
reform practices (Mayer, 1999). Likewise, analyses of student work provide very 
important information about student performance but do not directly assess or draw 
attention to the opportunity that a student has in the classroom to produce high- 
quality work. Classroom observations are the most direct way to measure 
instructional quality, but these are expensive and time-consuming to conduct. 

The purpose of this report is to present the results so far of work developing a 
method to describe and assess classroom practice that has high technical quality 
(reliability and validity), provides relevant and useful information to teachers and 
school reform program leaders, and is efficient enough to be used in large-scale 
evaluation settings. This research was developed from the strand of research at 
CRESST focused on the development of effective educational indicators. Specifically, 
this report presents data from two years of research investigating the use of 
teachers' assignments as an indicator of classroom practice in urban schools 
participating in a large-scale; comprehensive reform effort. 

Although this method is not intended necessarily to be restricted to a single 
subject area, our work so far has centered on language arts and student literacy. The 
first year of this research, led by Pamela Aschbacher, focused on developing and 
piloting the use of language arts assignments as an indicator of classroom practice. 
In her report of the findings from the first year of research (Aschbacher, 1999), she 
specifically addresses the usefulness of the assignment ratings, the relationship 
between the quality of the assignments and student work, and teacher interview 
data. She also reports analyses focused on the technical quality of the ratings and the 
feasibility of the method for use in large-scale settings. 

This report presents findings from the second year of the study (Clare, 1999; 
Clare, Pascal, Steinberg, & Valdes, 2000). The data collection strategy for this strand 
of research was expanded to include classroom observations. Further analyses 
regarding the reliability of this method with a larger sample of teachers and 
assignments are reported. The relationship between the quality of classroom 
assignments and observed instruction and the relationship between these 
assignments and the quality of student work are investigated as well. The results of 
additional generalizability and decision studies also are reported to further 
investigate the potential feasibility of this method for use in large-scale evaluations. 
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Importance of the Research 

An indicator is a statistic that measures outcomes or important dimensions of a 
system in comparison with a standard over time (Smith, 1988). The purpose of 
indicators is to describe the relative functioning of a system and point the way 
toward improving that system. Developing indicators of classroom practice thus is 
important for directing attention to dimensions of practice that are germane to 
student learning, as well as for monitoring and supporting diverse school reform 
efforts (Linn & Baker, 1998). Because of limitations in available methodologies for 
measuring classroom practice, the nature and quality of students 7 learning 
environments have existed as a "black box" in many large-scale evaluation designs. 
Reform monies are expended and student outcomes are measured, but little 
information is collected on a broad scale regarding the quality of instructional 
practice. It is imperative that indicators of classroom practice be developed that 
accurately describe the nature of students' learning environments and are sensitive 
enough to describe different aspects of practice, some of which may be differentially 
influenced by reform efforts (Spillane & Zeuli, 1999). This is key to providing high- 
quality formative feedback that helps diverse reform programs focus their efforts so 
that they more effectively benefit students. This is important as well for providing 
formative feedback to teachers and administrators on ways to improve the quality of 
learning and instruction at their schools, and for supporting teachers' reflection on 
and self-evaluation of their practice (Aschbacher, 1999; Newmann, Lopez, & Byrk, 
1998). 

Theoretical Perspectives on Defining Quality of Classroom Practice 

The framework for defining and measuring the quality of students' learning 
environments is rooted in sociocultural theory. Sociocultural theory proposes that 
development is rooted and unfolds in social contexts and that successful instruction 
hinges on the extent to which novices are given opportunities to engage in 
meaningful, goal-directed activities that "scaffold" student understanding and build 
on their funds of knowledge (Rogoff, 1990; Vygotsky, 1978). Research indicates that 
teachers often have difficulty maintaining high standards for student achievement, 
as reflected in assignment tasks that do not support the attainment of higher order 
thinking skills or require students to engage with meaningful content material 
(Newmann et al., 1998). Students often are not given the opportunity to actively 
participate in classroom discussions that build on and expand their thinking (Tharp 
& Gallimore, 1988) and do not receive meaningful feedback about their performance 
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and participation (Olson, 1990; Olson & Raffeld, 1987; Schunk & Swartz, 1993). 
Additionally, teachers' curriculum and instruction decisions often appear to be 
driven by "activity for activity's sake" rather than by clear goals for desired student 
outcomes (Duffy, 1981). This lack of coherence is further reflected in assessment 
criteria that are often not aligned with standards or goals for students (Aschbacher, 
1994; Aschbacher & Herman, 1991; Aschbacher & Rector, 1996). The quality of 
classroom practice in this study is thus defined in part by the degree to which 
students engage in complex thinking and use content knowledge in lessons and 
assignments, and by the degree to which teachers' goals are focused on student 
learning and are aligned with tasks and assessment criteria. We also examine 
students' opportunities to engage in classroom discussions and receive informative 
instructional feedback. 

Research Questions 

This report first explores the reliability and independence of the classroom 
assignment ratings. Evidence of the validity of using classroom assignments as an 
indicator of classroom practice also is investigated by comparing assignment ratings 
to ratings of classroom observations and to the quality of students' written work. 
The feasibility of using classroom assignments as an indicator of classroom practice 
in large-scale evaluation settings is examined in terms of the number of assignments 
and raters potentially needed to provide a consistent estimate of quality. The types 
of assignments that might provide the best estimate of instructional quality also are 
investigated. The specific research questions addressed in this study are as follows: 

1. How reliable are the classroom assignment rating scales? 

2. How independent are the classroom assignment rating scales? 

3. What is the relationship between classroom assignment ratings and other 
indicators of instructional quality (i.e., classroom observations and students' 
written work)? 

4. How many assignments and raters are needed to obtain a consistent 
estimate of the quality of classroom practice? 

5. What types of assignments might provide the best estimates of quality of 
classroom practice? 




Methods 



Sites and Participants 

This research is being conducted in a subsample of urban schools participating 
in a large-scale school reform initiative. Third-grade teachers (N = 12) and seventh- 
grade teachers (N = 12) were targeted for study in 4 elementary and 4 middle 
schools (N = 8 schools). The average number of years teachers had been teaching 
was approximately 11 years, with a range from 2 to 28 years. Table 1 presents the 
demographic characteristics for the elementary and middle schools chosen for in- 
depth study. 

Procedures 

Classroom assignments and student work. Teachers (N = 24) submitted four 
language arts assignments. These assignments were "typical" writing, "typical" 
reading comprehension, content area writing (elementary school only), and a 
"challenging" major project. Teachers also were asked to complete a one-page 
information sheet for each assignment and to submit four samples of student work 
for each assignment that they considered to be of "medium" quality and "high" 
quality. The teacher assignment materials (notebook, cover sheets, consent forms, 
etc.) were distributed in the fall and collected in the winter and spring (see 
Appendix A). 



Table 1 

Demographics and SAT-9 Scores for Elementary and Middle Schools (N = 8) 





Elementary schools 
<* = 4) 


Middle schools 
(«= 4) 




Mean % 


% Range 


Mean % 


% Range 


Enrollment by ethnicity 










Asian 


7.5 


0.3-27.0 


3.0 


0.7 -7.0 


African American 


12.0 


1.4-20.0 


18.0 


1.4-35.5 


Latino 


63.0 


34.0 - 92.1 


61.0 


41.0-92.7 


White 


14.5 


3.8 - 37.3 


13.0 


3.8-31.9 


Other 


3.0 


1.0 -7.0 


5.0 


1.3-17.0 


English language learner 


86.8 


50.4 - 82.2 


40.4 


30.0 - 60.2 


Free/reduced lunch 


89.9 


86.7-93.8 


72.7 


56.6-80.0 


1998-99 SAT-9 scores at or about 50th 
NPR in reading for Grades 3 and 7 


24.0 


17.0 - 32.0 


24.5 


15.0-36.0 
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Observations. These same teachers, plus two additional teachers who did not 
return the classroom assignment notebooks, were observed twice during the year in 
fall and winter ( N = 26). Observations lasted for one class period and were of a 
"typical" language arts lesson. Before each observation, we contacted principals and 
asked them to suggest dates and times when we could visit teachers' classrooms. We 
then contacted teachers to confirm that these dates and times were convenient for 
them. 

Interviews. The third- and seventh-grade teachers ( N = 26) were briefly 
interviewed about their lessons at each observation point. These interviews were 
approximately 15 minutes long and focused on the observed lesson activities (e.g., 
goals for the lesson, context, specific student needs, etc. (see Appendix B for the 
observation and interview protocols). 

Measures 

Classroom assignments. As described in the introduction of this report, our 
criteria for looking at the quality of classroom assignments (see Appendix C) are 
based on a sociocultural theoretical framework and embed a standards-based 
approach to curriculum and teaching. We also based our criteria on research that 
indicates that teachers do not always maintain high standards for student 
achievement or hold clear goals for student learning outcomes. Additionally, prior 
research has found teachers can have difficulty aligning their assessment criteria 
with standards and goals for students. Based on this research, we used a 4-point 
scale (1 = poor, 4 = excellent) to rate the following six dimensions of quality for each 
assignment (see Clare et al., 2000, pp. 2-3). 

• Cognitive challenge of the task. This dimension describes the level of 
thinking required of students to complete the task. Specifically this 
dimension describes the degree to which students have the opportunity to 
apply higher order reasoning and engage with academic content material. 
For example, an assignment given a high score for cognitive challenge 
might require students to synthesize ideas, analyze cause and effect, and/or 
analyze a problem and pose reasonable solutions using content-area 
knowledge (e.g., comparing themes from different books, etc.). An 
assignment given a low score on this dimension, in contrast, might require 
students only to recall very basic, factual information (e.g., "What color is 
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the car?") or to write on a topic requiring no academic content knowledge 
(e.g., a fan letter to a movie star). 

• Clarity of the teacher's goals for student learning. This dimension is 
intended to describe how clearly a teacher articulates the specific skills, 
concepts, or content knowledge students are to gain from completing the 
assignment. The primary purpose of this dimension is to describe the 
degree to which an assignment could be considered a purposeful, goal- 
driven activity focused on student learning. An assignment given a high 
score on this dimension would have goals that were very clear, detailed, 
and specific as to what students are to learn from completing the 
assignment. It would also be possible to assess whether or not students had 
achieved these goals. For example, the goals stated for one elementary 
school classroom assignment given a high score on this dimension were that 
students "learn the concepts of life cycle and food chain (e.g., whether an 
animal is prey, predator, or both; habitat, and the idea that different animals 
live on different continents)." In contrast, the stated goals for another 
assignment were that students "delete the information and improve the 
information needed" on a worksheet. This assignment was given a low 
score for clarity of teacher's goals since it was not clear what specific 
concepts or skills students were to learn from completing this task. 

• Clarity of the grading criteria. The purpose of this dimension is to assess 
the quality of the grading criteria for the assignment in terms of their 
specificity and potential for helping students improve their performance. 
How clearly each aspect of the grading criteria is defined is considered in 
the rating, as well as how much detail is provided for each of the criteria. 
An assignment given a high score for this dimension would have grading 
criteria in which the guidelines for success were clearly detailed and 
provided a great deal of information to students for what they needed to do 
to successfully complete the task (e.g., "To have a fully developed incident 
that tells a story, your writing will include dialogue, movement, gestures, 
names of people, and sensory details"). An assignment that was given a low 
score for this dimension, in contrast, would have unclear and nonspecific 
grading criteria. For example, one assignment scored as such was reported 
by the teacher to be graded "by looking at story development and 
creativity — completely subjective." 




7 



11 



• Alignment of learning goals and task. This dimension focuses on the 
degree to which a teacher's stated learning goals are reflected in the design 
of the assignment tasks students are asked to complete. Specifically, this 
dimension attempts to capture how well the assignment appears to promote 
the achievement of the teacher's goals for student learning. An assignment 
given a high score on this dimension would involve tasks and goals that 
overlapped completely. For example, the learning goals for one such 
assignment were that students develop summary skills and learn to 
distinguish between relevant and irrelevant facts and details, and the task 
called for students to summarize the important points of a story. In contrast, 
the learning goals for an assignment given a low score on this dimension 
were that seventh-grade students recall information, develop 
understanding, and remember important facts. The task, however, only 
required students to write down facts that were written on the board. This 
assignment was given a low score for this dimension since copying 
information does not "match" the learning goals for development of 
memory skills or understanding. 

• Alignment between the goals and grading criteria. This dimension is 
intended to describe the degree to which a teacher's grading criteria 
support the learning goals. In other words, this dimension focuses on the 
degree to which a teacher assesses students on the skills and concepts they 
are intended to learn through the completion of the assignment. Also 
considered in this rating is whether or not the grading criteria include 
extraneous dimensions that do not support the learning goals, as well as the 
appropriateness of the criteria for supporting the learning goals. An 
assignment given a high score for this dimension would have goals and 
grading criteria that overlapped completely. An assignment given the 
lowest score on this dimension, in contrast, would have grading criteria that 
did not support the learning goals. For example, the learning goals for one 
such assignment were that students learn to use "proper business language 
[and] business letter format." In describing the grading criteria, however, 
the teacher commented that students were not penalized for their use of 
slang if the letter was a "fun letter to a movie star." While it may be 
appropriate to use slang in a letter to a movie star, this assignment was 
given a low score on this dimension since the assessment criteria do not 
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match the stated goals that students learn to use formal business language 
and apply a business letter format. 

• Overall quality of the assignment. This dimension is intended to provide a 
holistic rating of the quality of the assignment based on its level of cognitive 
challenge, the specificity and focus of the learning goals, the clarity of the 
grading criteria, the alignment of the learning goals and the assignment 
task, and the alignment of the learning goals and the grading criteria (see 
Appendix C for further description of these dimensions based on the 
assignments collected in elementary and middle schools). 

Each assignment was scored by three independent raters on these dimensions. 
Raters (N = 3) underwent approximately two weeks of training before scoring the 
assignments. This training included scoring assignments collected from non-sample 
teachers individually and as a group, and selecting anchor papers by scale point and 
dimension to calibrate ratings and refine the rubric. Overall exact scale-point 
agreement for the classroom assignment scales was 84% (see Appendix D). 

Student work. We scored student work from the writing assignments (a final 
writing project with earlier drafts). Student writing was rated by two bilingual raters 
using three standards-based scales measuring organization, content, and MUGS (i.e., 
mechanics, language use, grammar, and spelling). These scales were from the 
Language Arts Project rubric developed by CRESST at UCLA, in partnership with 
LAUSD and United Teachers-Los Angeles (Higuchi, 1996). Each of these dimensions 
was rated on a 4-point scale (1 = poor, 4 = excellent ; see Appendix E). Both raters had 
extensive experience scoring student work using these rating scales. Interrater 
reliability was assessed by having both raters score a subset of 20% of the 
assignments chosen at random. Overall exact scale-point agreement for these scales 
was 81%. 

Classroom observations. Observations were conducted by experienced 
research staff and graduate students. Researchers wrote detailed field notes 
describing the classroom, lesson activities, and the interactions between the teacher 
and the students. The length of each activity (measured in number of minutes) and 
the number of students involved in each of the observed lesson activities also were 
recorded, and each activity was categorized according to social organization, 
behavior of the teacher and students, resources in use, and language arts content. 
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A 4-point scale (1 = poor, 4 = excellent) was used to describe the overall quality 
of the observed lesson for the following eight dimensions. 

• Challenge of the lesson activities. This dimension is intended to describe 
the level of thinking required of students to participate in the observed 
lesson activities (e.g., the degree to which students had the opportunity to 
think critically; predict, analyze, and synthesize information; and engage 
with substantive content material). 

• Quality of classroom discussions. The quality of the classroom discussion 
or instructional conversation also is considered as a critical dimension of 
classroom practice. This dimension captures the extent to which the teacher 
provided students with the opportunity to learn through and engage as 
partners in meaningful classroom discussions. This includes both the nature 
of a teacher's questions and the degree to which student contributions are 
extended and built on, as well as the amount of time spent in discussion. 

• Level of student participation in classroom discussions. This dimension is 
intended to describe the percentage of students who engaged in classroom 
discussions. 

• Quality of instructional feedback. This dimension describes students' 
opportunity to receive information about their performance and progress 
toward learning goals and the degree to which this feedback appears to 
support learning. The accuracy, substance, specificity, and helpfulness of 
the teacher's feedback are considered in the ratings, as well as the amount 
of feedback the teacher provided to students during the observed lesson. 

• Level of student engagement in the lesson. This dimension is intended to 
capture the level of student engagement in the observed lesson activities. 
Specifically this dimension describes the percentage of students who 
appeared to be on task and participating in the lesson activities. 

• Lesson implementation/classroom management. This dimension is 
intended to describe the degree to which a teacher effectively carries out the 
lesson activities. This scale focuses on a teacher's classroom management 
skills, including the amount of time spent on transitions from one activity to 
another or procedural tasks, and how disruptive or distracting student 
behavior was handled. 
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• Clarity of the teacher's goals for the lesson. This dimension is intended to 
capture the degree to which a teacher is able to articulate the specific skills, 
concepts, or content knowledge students are to gain from participating in 
the lesson or lesson activities. This information is obtained from 
interviewing the teacher prior to observing the lesson. The primary purpose 
of this dimension is to capture the degree to which lessons could be 
considered purposeful, goal-driven activities focused on student learning 
versus "activity for activity's sake." In other words, this dimension attempts 
to differentiate between teachers who plan their lessons with clear and 
specific learning goals in mind and those who plan activities with no clearly 
defined learning objective. The quality of the activities themselves — which 
the goals may or may not describe — is not considered in this rating. 

• Alignment between goals and the lesson activities. This dimension 
attempts to capture the degree to which a teacher's stated goals for the 
lesson are reflected in the design of the learning activities. Specifically this 
dimension attempts to capture how well the learning activities promote the 
achievement of the teacher's goals for student learning (see Appendix F for 
more detailed description of these rating scales). 

Training for observers (N = 4) included coding videotapes as a group and 
observing in pairs in non-sample third- and seventh-grade classrooms at two 
different points during the year prior to observing in our sample schools. Reliability 
was assessed by comparing the scores for each possible pair of observers. Overall 
exact scale-point agreement was 77.5%. 

Analyses 

Descriptive statistics were used to describe the quality of teacher assignments, 
classroom observations, and student work. Cohen's kappa coefficients were 
calculated to investigate the proportion of agreement between raters after chance 
agreement was removed. Cronbach's alpha coefficients were calculated to estimate 
the internal consistency of the ratings (Abedi, 1996). Principal components analysis 
was used to explore the interrelationship of scale items and dimensions of quality 
practice for teacher assignment and classroom observation ratings, and to reduce the 
data. Correlation coefficients were computed to examine the relation of teacher 
assignment ratings to ratings of classroom observations and student work, and to 
investigate the interrelationship of the rating scales. A generalizability study was 
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conducted to investigate the consistency of our classroom assignment ratings, and a 
decision study was conducted to investigate the number of assignments and raters 
most likely needed to obtain a consistent rating of quality classroom practice. 



Results 

In this section, we provide a brief description of the quality of the classroom 
assignments in our sample as a context for our findings. Results then are presented 
organized around the different research questions. Specifically, we report data based 
on analyses of teachers' assignments, student work, and classroom observations to 
explore the technical quality of our method and the feasibility of using teachers' 
assignments as an indicator of classroom practice in large-scale evaluation settings. 

Quality of Classroom Assignments 

As illustrated in Table 2 and Table 3, the quality of the different types of 
teachers' assignments we collected in third- and seventh-grade classrooms ( N = 24) 
tended to be fairly basic. This is especially true with regard to the level of cognitive 
challenge and the alignment of the teachers' goals for student learning with their 
assessment criteria. While we saw examples of excellent assignments, the majority of 
the assignments were scored a 2 on the majority of the rating dimensions at both 
levels of schooling. (See Appendix C for examples of assignments that were given 
high and low scores for the different dimensions and the frequencies for each scale 
point for each assignment type.) 



Table 2 

Description of Elementary School Teacher Assignments (N = 48) 





Reading 
comprehension 
(« « 12) 
M(SD) 


Writing 
(« « 12) 
M(SD) 


Content area 
writing 
(« = 12) 
M(SD) 


Challenging 
(n = 12) 
M(SD) 


Level of cognitive challenge 


1.94 (.65) 


1.82 (.43) 


1.91 (.34) 


2.48 (.50) 


Clarity of learning goals 


2.15 (.50) 


2.03 (.41) 


1.76 (.50) 


2.21 (.40) 


Clarity of grading criteria 


1.83 (.79) 


2.06 (.83) 


1.90 (.57) 


2.27 (.98) 


Alignment of goals and task 


2.00 (.52) 


2.12 (.48) 


1.97 (.57) 


2.33 (.47) 


Alignment of goals and grading 
criteria 


1.57 (.75) 


1.70 (.46) 


1.50 (.59) 


1.87 (.74) 


Overall quality 


1.82 (.60) 


1.82 (.50) 


1.76 (.45) 


2.27 (.47) 



Note. Items were scored on a 4-point scale (1 = poor, 4 = excellent). 
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Table 3 

Description of Middle School Teacher Assignments (N = 46) 





Reading 
comprehension 
<« = 11) 

M (SD) 


Writing 
(n = 24) 
M (SD) 


Challenging 
(n = 11) 

M (SD) 


Level of cognitive challenge 


1.97 (.74) 


2.14 (.54) 


2.39 (.59) 


Clarity of learning goals 


1.82 (.38) 


2.00 (.50) 


2.12 (.65) 


Clarity of grading criteria 


1.64 (.50) 


2.00 (.78) 


2.10 (.59) 


Alignment of goals and task 


2.00 (.42) 


2.12 (.50) 


2.06 (.59) 


Alignment of goals and grading criteria 


1.48 (.48) 


1.80 (.74) 


1.77 (.39) 


Overall quality 


1.76 (.42) 


1.97 (.54) 


2.21 (.54) 



Note. Items were scored on a 4-point scale (1 — poor, 4 — excellent). 



How Reliable Are the Classroom Assignment Rating Scales? 

As part of our investigation of the technical quality of our indicator system, we 
looked at interrater reliability for the evaluative scales — the degree to which 
different people can independently look at the same phenomenon (in this case 
teachers' assignments) and agree on a score. We also investigated the internal 
consistency of the classroom assignment scales. As illustrated in Table 4, kappa 
coefficients for each dimension for each assignment type were statistically 



Table 4 

Reliability of Classroom Assignment Rating Scales Across Assignment Types (N = 37 Teachers) 









% of exact scale- 


Scale 


Kappa 3 


Alpha 


point agreement^ 


Challenge of the task 


.45 - .56 


.80 - .86 


85.3 


Clarity of the teacher's goals for student learning 


.34 - .56 


.74 -.85 


86.1 


Quality of assessment criteria 


.42 - .59 


.84 - .91 


84.0 


Alignment of goal and task 


.27 - .47 


.68 -.84 


80.1 


Alignment of goal and assessment criteria 


.43 - .53 


.84 - .90 


82.7 


Overall quality of task 


.38 - .54 


.73 - .85 


86.4 



Note. These analyses include 6 additional elementary school teachers and 7 additional middle 
school teachers from the first year of the study. Their parallel assignments were re-scored by the 
same pool of raters to reduce coding bias and to measure the clarity of teachers' goals for student 
learning (a new scale that had been added to the rubric). 

a Kappa coefficients for each assignment are significant at p < 0.01 and p < 0.001. 
b Calculated as percent of exact scale-point agreement between two raters. 
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significant (k = .35, p < .01 to k = .59, p < .001). Alpha coefficients for each dimension 
for each assignment type also were acceptable (a = .68 to .91), though somewhat low 
for the alignment of goal and task for the reading comprehension assignment (a = 
.68). Percent of exact scale-point agreement between at least two raters also was 
acceptable for each dimension and ranged from 80.1% for the alignment of goal and 
task to 86.4% for the overall quality of the assignment task. We concluded from 
these analyses that our ratings of classroom assignments demonstrated an 
acceptable level of reliability. 

How Independent Are the Classroom Assignment Rating Scales? 

In this section we explore the interrelationship of the classroom assignment 
rating scales. Evaluating large-scale reform efforts can be quite costly, and so it is 
imperative that measurement tools be as efficient and streamlined as possible. The 
purpose of examining the interrelationship of the rating scales is to reduce 
redundancy in our rating scheme by investigating whether certain scales may be so 
highly correlated with one another that some scales could be eliminated. 

As illustrated in Table 5 the correlation matrix for the combined assignments 
indicates that most of the scales are statistically significantly associated with one 
another. This is especially true for the scale measuring the overall quality of the 
assignments (r = .43, p < 0.001 to r = .72, p < 0.001), which makes sense given that 
this scale is intended to represent a holistic indicator of quality. This suggests that it 
might be possible to holistically rate assignment quality. At this stage in our 
research, however, we are reluctant to draw definitive conclusions about which 

Table 5 



Interrcorrelation of Classroom Assignment Ratings Across Assignment Types 





Cognitive 

challenge 


Clarity of 
goals 


Grading 

criteria 


Alignment 
goals/ task 


Alignment 

goals/grade Overall 


Cognitive challenge 


1.0 










Clarity of goals 


0.25** 


1.0 








Grading criteria 


0.35*** 


0.16 


1.0 






Alignment of goals/ 
task 


0.34*** 


0.36*** 


0.42*** 


1.0 




Alignment of goals/ 
grade 


0.34*** 


0.28*** 


0.68*** 


0.59*** 


1.0 


Overall 


0.72*** 


0.43*** 


0.53*** 


0.57*** 


0.52*** 1.0 



**p < 0.01. ***p < 0.001. 
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scale(s) to eliminate. It would be important to conduct analyses with coders who are 
less familiar with the instrument in order to determine whether the other scales 
continue to be associated with the one scale measuring the overall quality of the 
assignment (see Appendix G for additional tables measuring the interrcorrelation of 
ratings by assignment types 1 ). Additionally, retaining the different subscales might 
also be important in terms of helping to communicate the different dimensions that 
are important to look at when assessing the quality of classroom assignments, and 
could help provide more specific feedback to teachers and school reform programs. 

What Is the Relationship Between the Classroom Assignment Ratings and Other 
Indicators of Instructional Quality (i.e.. Classroom Observations and Student 
Work)? 

In this section we look for evidence of the construct validity of our method by 
comparing our ratings of the assignments to other indicators of instructional quality. 
The purpose of this is to assess the degree to which the classroom assignment 
ratings provide us with meaningful and appropriate information about students' 
learning environments that is commensurate with other measures of quality 
practice. We first investigate the pattern of relationships between the quality of 
classroom assignments and the quality of instruction observed in teachers' 
classrooms. We then investigate the relationship between the classroom assignment 
rating scales and the quality of student work. 

To investigate the relationship between the quality of classroom assignments 
and observed instruction, we first conducted exploratory factor analyses to reduce 
the data and to examine the underlying dimensions of the ratings scales. These 
analyses revealed two underlying dimensions in the classroom observation data. 
The first factor based on the classroom observation ratings included variables that 
measure aspects of constructivist-type practice — the quality of the instructional 
conversation, student participation in the instructional conversation, the challenge of 
the lesson, and quality of the teacher's instructional feedback to students. The 
second factor measured how well the lesson was implemented in terms of classroom 
management, student engagement in the lesson, focus of the teacher's goals on 
student learning, and alignment between the teacher's goals and lesson activities. 
Factor analysis with teacher assignment data, in contrast, revealed a single factor for 



1 We also investigated the interrcorrelation of the classroom assignment ratings by level of schooling. 
These analyses did not show a unique pattern of results separate from assignment type. 
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the rating scales which is not surprising given how highly correlated these scales 
were with one another (see Table 6 and Table 7). 

After the data had been reduced into factors, ratings from the observed lessons 
from winter 1999 (the lesson closest to when the assignments were given to 
students) were correlated with the teacher assignment ratings (from spring 1999). 
These analyses revealed that the first classroom observation factor measuring 
elements of constructivist practice was significantly associated with the quality of 
teachers' assignments (r = .57, p < 0.01). The second observation factor, in contrast, 
measuring the quality of the lesson implementation, was not significantly 



Table 6 



Exploratory Factor Analysis of Classroom Observation Variables (N = 26 Teachers) 



Variables 


Factor 1: 
Constructivist 
practices 


Factor 2: 
Lesson 

implementation 


Quality of instructional conversation 


.93 


-.09 


Challenge of the lesson 


.82 


.07 


Student participation in instructional conversation 


.76 


.02 


Quality of instructional feedback 


.38 


.34 


Student engagement in lesson activity 


-.09 


.88 


Quality of classroom management 


-.09 


.87 


Clarity and focus of the teacher's goals on student learning 


.15 


.73 


Alignment of goals and activity 


.26 


.57 


Eigenvalue 


4.06 


1.08 


Percent of variance explained 


50.8 


13.5 



Note. Boldface type indicates the variables that comprise each factor. 




Table 7 



Exploratory Factor Analysis of Teacher Assignments (N = 24 Teachers) 



Variables 


Factor 1: 

Teacher assignments 


Overall quality of task 


.93 


Alignment of goal and task 


.90 


Alignment of goal and assessment criteria 


.88 


Quality of assessment criteria 


.81 


Challenge of the task 


.78 


Clarity of teacher's goals for the task 


.69 


Eigenvalue 

Percent of variance explained 


4.19 

69.9 
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associated with teachers' assignments ( r = .03). The relationship of the observation 
factors to each other also was investigated. This analysis revealed that these factors 
were significantly associated with each other (r = .50, p < 0.01). 

Our analyses indicate that ratings of classroom assignments yield similar 
estimates of quality obtained from classroom observations with specific regard to 
elements of constructivist practice (i.e., challenge of the lesson activities, level in 
instructional discourse, and quality of teachers' feedback to students during the 
lesson). 

To further look for evidence of the validity of using teachers' assignments as an 
indicator of classroom practice, we examined the relationship between the quality of 
classroom assignments and the quality of student work. To do this we correlated the 
different dimensions of the teacher assignment scales to the quality of student work. 
This analysis revealed that the quality of teachers' assignments was associated with 
the quality of student writing as assessed by the scales measuring writing content 
and organization. The teacher assignment rating scales also were for the most part 
significantly associated with students' command of writing mechanics (MUGS), 
though the strength of this relationship tended to be weaker overall (see Table 8). It 
appears that the quality of classroom assignments is statistically significantly 
associated with the quality of student work, though it is important to note that our 
analysis does not directly test for direction of influence — or a causal 
relationship — between quality of assignments and quality of student work. Other 
factors, such as the quality of supporting instruction around assignments, are 
certainly important in terms of fostering high-quality student work. 



Table 8 

Relationship of Classroom Writing Assignments and Student Writing in the 1998-99 
Evaluation Year (N = 24 Teachers) 





Student writing scales 
Content Organization 


MUGS 


Challenge of the task 


.30*** 


34*** 


. 27 ** 


Clarity and focus of teacher's goals 


.22* 


.28** 


.15 


Quality of assessment criteria 


29*** 


.29** 


.19* 


Alignment of goal and task 


.28** 


.33*** 


.17 


Alignment of goal and assessment criteria 


.30*** 


.36*** 


.22* 


Overall quality of task 


.31*** 


34*** 


■18(*) 



*p< 0.05. **p<0.01. ***p< 0.001. 
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How Many Assignments and Raters Are Needed to Obtain a Consistent Estimate 
of the Quality of Classroom Practice? 

In this section we explore the number of raters and assignments needed to 
obtain a stable estimate of quality. To explore this issue we first conducted a 
generalizability study to investigate the quality of design in terms of yielding 
consistent estimates of classroom practice. This analysis revealed a G-coefficient of 
.91 and .87 for our design that used three raters to rate four assignments collected in 
elementary and middle schools respectively. Looking at the estimated variance 
components of the teacher assignment ratings, we found (as hoped) that most of the 
variation in the ratings was accounted for by differences across teachers and not by 
differences across raters. Interestingly, we also found that there was little variation 
across the different types of assignments collected from teachers (see Table 9). 

To further explore this issue we conducted a decision study in order to estimate 
generalizability coefficients for varying numbers of assignments and raters. As 
illustrated in Table 10, the decision study conducted with our current data estimated 
that the minimum number of assignments and raters needed to yield a consistent 
measure of quality was 2 each at both the elementary (G = .87) and middle school 
(G = .82) levels. 

Table 9 



Estimated Variance Components and Percent of Variance Explained by Teacher, Assignment Type, 
and Rater (N = 37 Teachers) 





Elementary school 
(n = 18) 


Middle school 

(« = 19 ) 




Variance 


% of 


Variance 


% of 




components 


variance 


components 


variance 


Teacher 


.1757 


.7426 


.1748 


.6356 


Assignment type 


.0049 


.0207 


.0098 


.0356 


Rater 


.0025 


.0102 


.0117 


.0425 


Teacher by assignment type 


.0000 


.0000 


.0000 


.0000 


Teacher by rater 


.0535 


.2261 


.0782 


.2844 


Assignment type by rater 


.0000 


.0000 


.0005 


— 


Teacher by assignment type by rater 


.0000 


.0000 


.0000 


.0000 



Note. These analyses include 6 additional elementary school teachers and 7 additional middle 
school teachers from the first year of the study. Their parallel assignments were re-scored by the 
same pool of raters to reduce coding bias and to measure the clarity of teachers' goals for student 
learning (a new scale that had been added to the rubric). 
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Table 10 



Estimated G-Coefficients Based on the Number of Assignments and 
Raters (N = 48) 



Number of 
assignments 


Number of 
raters 


G-coefficient: 
Elementary school 


G-coefficient: 
Middle school 


4 


3 


.91 


.87 


2 


2 


.87 


.82 


2 


1 


.77 


.69 



What Types of Assignments Might Provide the Best Estimates of Quality of 
Classroom Practice? 

Ultimately the question of which assignments are best to collect depends on the 
purpose and focus of the individual evaluation or research project. For our 
purposes, however, our concern is centered on figuring out which assignments 
might provide the best estimates of the quality of students' learning environments. 
To investigate this we examined the relationship between the different assignment 
types and classroom observations. This revealed that the typical reading 
comprehension assignments (r = .56, p < 0.01) and typical writing assignments 
(r = .68, p < 0.001) were statistically significantly associated with the quality of 
observed instructional practice with regard to constructivist practices. The typical 
reading comprehension assignment also was associated with the variables 
measuring the quality of lesson implementation (r = .41, p < 0.05). The content area 
writing assignment (given at the elementary school level only) and the challenging 
assignment in contrast were not associated with the quality of observed instruction 
(see Table 11). We concluded from this analysis that everyday or typical reading 
comprehension and writing assignments likely provide the best view on the quality 
of students' learning environments or classroom practice. 



Table 11 

Correlation of Assignment Types to Observed Instruction (N = 24 teachers) 



Assignment type 


Observation factor 1: 
Constructivist practices 


Observation factor 2: 
Implementation 


Typical reading comprehension 


0.56** 


0.41* 


Typical writing 


0.68*** 


-0.02 


Content area writing (elementary schools only) 


0.10 


-0.06 


Challenging 


0.26 


-0.19 



*p < 0.05. * *p< 0.01. ***p < 0.001. 
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Summary of Findings and Limitations of the Research 

The results of our study indicate that we had good reliability for the classroom 
assignment ratings. These results suggest that it is possible for raters to agree on and 
consistently rate the quality of assignments using our dimensions. We looked for 
evidence of the construct validity of our method by examining the relationship 
between the classroom assignment ratings and other indicators of quality 
instructional practice. We found that ratings of classroom assignments yielded 
estimates of quality similar to ratings obtained from classroom observations. 
Specifically we found that the quality of classroom assignments was significantly 
associated with elements of constructivist practice observed during the lesson (i.e., 
challenge of the lesson activities, level in instructional discourse, and quality of 
teachers' feedback to students). 

Commensurate with findings from the first year of the study, our analyses also 
revealed that the quality of classroom assignments was statistically significantly 
associated with the quality of student work (see Aschbacher, 1999). The results of 
this analysis, which replicate findings from the first year of the study with 
additional teachers and assignments, provide additional support for using teachers' 
assignments as an indicator of classroom practice. As mentioned before, however, it 
is important to bear in mind that we did not directly test for direction of 
influence — or a causal relationship — between the quality of assignments and the 
quality of student work in our analyses. In other words, we cannot conclude that 
being exposed to high-quality assignments guarantees that students will be able to 
produce high-quality work themselves. Students also need supportive classroom 
instruction and feedback in order to learn the skills and concepts required to 
complete high-quality assignments. We therefore regard classroom assignment 
quality to be a "necessary though not sufficient condition" for student success 
(Newmann et al., 1998, p. 30). More research is necessary to determine the specific 
contribution high-quality assignments make to increased student academic 
achievement, independent of classroom instruction. 

Finally, we continued to investigate the feasibility of our method for use in 
large-scale evaluation settings. Also commensurate with the decision study 
conducted during the first year of the study (see Aschbacher, 1999), our results 
showed that our design collecting four assignments from teachers and using three 
raters yielded a consistent estimate of quality. Additional work that should be 
undertaken in the area includes training additional raters and monitoring their 
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scores as a way to explore the amount and type of training necessary to reliably 
score assignments. This is an important piece of information with regard to the 
feasibility and cost-effectiveness of using this method to evaluate a large number of 
classrooms. In this vein we also investigated the minimum number of raters and 
assignments needed to obtain a consistent estimate of quality. Our analyses 
estimated that two assignments from teachers and two raters could yield a 
consistent estimate of quality at both the elementary and middle school levels. 

We then examined the relationship between the type of assignment collected 
and the quality of observed instruction in order to determine which assignments 
might yield the best estimate of quality classroom practice. We found that the typical 
writing assignment and the typical reading comprehension assignment were 
significantly statistically associated with the quality of observed instruction. We 
concluded from this that these assignments might provide the best view on the 
quality of students' classroom learning environments. Additional research is 
necessary to further investigate whether a design of two assignments and two raters 
would in actuality yield a consistent estimate of quality and whether typical writing 
and reading comprehension assignments continue to be associated with the quality 
of observed language arts instruction. 

As a final caveat, across the board our findings are limited by the fact that we 
had a small sample size. More than this, however, the power and generalizability of 
our analyses potentially are limited by a restriction in range in our sample in terms 
of classroom learning environments. All of the schools in our present sample are in 
urban areas and serve primarily poor and minority students. This raises the question 
of whether the same patterns of results would hold if the sample included other 
types of schools (e.g., schools serving more privileged students). These limitations 
aside, however, the results of our research to date indicate that our method shows 
considerable promise as a measure of quality classroom practice in terms of 
technical quality, usefulness, and potential for use on a large-scale basis. We will be 
continuing our investigation of this method during the 1999-2000 academic year 
(our third and final year of data collection in these sites) and will report the results 
of this work in a future CRESST report. 

Next Steps and Directions for Future Work 

Building on the results of analyses presented here, we recommend the 
following steps and directions for future research. 
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• Increase the number of teachers in the sample. 

• Expand the sample to include a more diverse range of schools. 

• Use both quantitative and qualitative methods to better assess change over 
time in the quality of assignments with a larger sample of teachers. 

• Expand the pool of raters in order to investigate the amount and type of 
training it takes to reliably and consistently score assignments. 

• Develop more refined scoring guides and anchor papers to facilitate the use 
of this method. 

In conclusion, in addition to creating a useful and effective measure of 
classroom practice, we hope as well that our framework will be useful to teachers in 
terms of reflecting on assignments with colleagues in grade-level or subject-level 
meetings or in the context of other professional development activities. Specifically, 
we hope that our framework might contribute to the improvement of teachers' 
instructional practices by helping teachers think about the kinds of skills their 
assignments promote, how well their assignments promote their learning goals, the 
clarity and specificity of their grading criteria, and the match between their grading 
criteria and their learning goals. We therefore suggest that future research include 
piloting the scoring framework with teachers in collaborative professional 
development settings (e.g.. Critical Friends Groups), in order to explore their 
perspectives on how useful the dimensions are for reflecting on the quality of their 
assignments. 
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APPENDIX A 



Teacher Assignment Materials 
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Directions for Collecting Assignments and Student Work 

Step-By-Step Process 



Due: February 1999 



Overview : 

Please collect 4 assignments with 4 samples of student work for each assignment. 
You will be asked to fill out a cover sheet for each assignment. Detailed instructions 
are given below. We want to describe the nature of the language arts tasks that 
students do, what's expected of them, what feedback they are given, and how 
grades are assigned. Our descriptions depend on what you tell us, so please be 
explicit and detailed so we can be as accurate as possible. Thanks. 

1. COLLECT THE FOLLOWING 4 ASSIGNMENTS. 

Between now and February, collect 4 of the assignments you give your third 
grade students, with selected examples of student work. Use assignments which 
ask students to do some individual written work. Do not create new 
assignments specifically for this study . Please collect the following types of 
assignments: 

• 3 typical in-class assignments with a written response (one of each of the 
following): 

• 1 reading comprehension or reading response assignment 

• For 3rd grade: 1 writing assignment in a content area such as social 
studies or science 

For 7th grade: 1 writing assignment of any type typical for your class 

• 1 writing assignment that includes a rough draft and final draft, with 
any written feedback given by peers or teachers (writers' workshop 
activities are fine) 

• 1 very challenging assignment or project with a written component 

Pick one of the most rigorous assignments involving language arts that 
you will give students any time between now and February of this school 
year. The assignment should elicit some high-level thinking and include 
some writing. It may include reading, oral report, or other activities. If 
this assignment has multiple steps, please submit only the written portion 
of the student work. 

If you are in doubt about what assignments are appropriate, just call us to discuss it. 

(continued) 
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2. FOR EACH OF THE 4 ASSIGNMENTS COPY 4 SAMPLES OF STUDENT 
WORK. 



• Choose two middle quality and two high quality pieces of student work 
from the same class. 

It is fine to choose different students' papers for the different assignments. 

We just need two middle and two high for each assignment. If there were no 
students who did high quality work on an assignment, attach a note 
explaining why you are not including any "High" pieces of student work. 

• Copy the four pieces of student work for each assignment. 

• Place an ID sticker over each student's name. (We prefer to receive student 
work without their names so as to protect their privacy). Please do not cover 
up any part of the student's work, your feedback, or grade. It is important for 
us to see the feedback comments or grades. If there is no clear area for the 
label, put it on the back of the work and cross out/ white out the student's 
name. 

• Note: The student ID labels for Assignment #1 are stapled to the 
pocket for Assignment #1, and so forth. 

• Place an M (Middle) or H (High) sticker on each student paper accordingly. 
These stickers are in the plastic sleeve immediately preceding the pockets for 
student work. 

• We would prefer to see student work from 3rd and 7th graders where 
possible. However, if you have students of another grade in your class and 
cannot find high or middle level work from your 3rd or 7th graders, record 
the grade level at the top of the student work if it is from a student who is not 
in 3rd or 7th grade (e.g., write "grade 2" at the top if you cannot find high 
level work from a 3rd grader and instead submit a high level work sample 
from a second grader in your 2/3 combination class). Call us if you are 
unsure. 



3. FILL OUT A COVER SHEET FOR EACH OF THE 4 ASSIGNMENTS. 

Fill out the enclosed Cover Sheets for Teacher Assignments in the pockets in this 
binder. There is a different cover sheet for each type of assignment, each on a 
different color of paper. 

• Please attach whatever will help us understand the assignment and 
accompanying student work, such as the following: 

- copy of the directions given to students (please be as explicit as possible), 

- grading rubric or guidelines, and 

- outline of the unit. 

• Place the cover sheet with attached papers and the 4 pieces of student work in 
the appropriate pockets in this binder. 
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General Information Form 
3rd Grade Teachers 



Please answer the following questions in January 1999. 

1. What grade level(s) are your students? Check all that apply. [ ] 2nd [ ] 3rd [ ] 4th 

2. How many years have you been teaching? years 

a. How many years have you taught 3rd grade? years 

3. How many students are enrolled in your class? 

4. Approximately what percentage of your students have been in your class since the beginning of 

the school year? % 

5. Please circle any of the following which describe your class: 

a. full bilingual b. modified bilingual c. SDAIE or sheltered English 

d. English only e. other (explain) 

6. Approximately what percent of your students are LEP (Limited English Proficient)? % 

a. In what language(s) do your LEP students receive language arts instruction? (Circle as many 
as apply) 

English Spanish Other 

b. Approximately what percent of your students have recently (within the past six months) 

been redesignated as Fluent English Proficient (RFEP)? % 

7. a. What is the range in reading level among your students? grade to grade 

b. At what grade level are most of your students currently reading? grade 

8. How similar is the language arts curriculum and instruction in your class to that of other teachers 
at your grade level in your school? (circle your answer) 

Not at all similar Somewhat similar Very similar 

1 2 3 4 5 

9. Is there anything else about your language arts class we should know when looking at the 
assignments and student work? 



Thanks so much! 
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Cover Sheet for Typical Reading Comprehension Assignment 

If you need more room to answer the questions, please use the back of this form or attach sheets as necessary. 

1. Describe the assignment below in detail or attach a copy of the assignment directions to this sheet. 

Be sure to tell us exactly what directions were given to students. 

Specify the title and type (e.g., poem, novel, textbook, etc.) and grade level of the reading material. If 
students are working in reading groups, specify which group was given this assignment. 



2. What concepts, skills, and/or processes did you expect the students to acquire from this assignment? 
i.e., what were your learning goals for the students? 



3. How does the assignment fit in with your unit or what you are teaching in your language arts class this 
month or this year? Is this an end-of-unit assessment? □ yes □ no How long did the task take? 



4. What type of help, if any, did students receive to complete the assignment? (Check all that apply.) 
Students received help or formative feedback from a □ teacher □ teacher's aide □ other students 
□ parents (e.g., help = substantive revision feedback from teacher or peers). Please explain: 



5. How was this assignment assessed? If there is a rubric, student reflection, etc., please attach it. 
If you are not attaching a rubric, please explain your criteria for grading the work (if graded). 
Did you share these criteria with students? [ ] yes [ ] no 



6. What criteria did you use to decide which papers are "M" middle papers and which are "H" high? 
(especially if work was not graded originally or if different from #5 above) 



7. Approximately what percent of students performed at the following levels on this assignment: 

% = good-excellent % = adequate % = not yet adequate 
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Cover Sheet for Typical Writing Assignment: Final and Rough Drafts 

If you need more room to answer the questions, please use the back of this form or attach sheets as necessary. 

1. Describe the assignment below in detail or attach a copy of the assignment directions to this sheet. 

Be sure to tell us exactly what directions were given to students. 

Specify the title and type (e.g., poem, novel, textbook, etc.) and grade level of any reading material. If 
students are working in reading groups, specify which group was given this assignment. 



2. What concepts, skills, and/or processes did you expect the students to acquire from this assignment? 
i.e., what were your learning goals for the students? 



3. How does the assignment fit in with your unit or what you are teaching in your language arts class this 
month or this year? Is this an end-of-unit assessment? □ yes □ no How long did the task take? 



4. What type of help, if any, did students receive to complete the assignment? (Check all that apply.) 
Students received help or formative feedback from a □ teacher □ teacher's aide □ other students 
□ parents (e.g., help = substantive revision feedback from teacher or peers). Please explain: 



5. How was this assignment assessed? If there is a rubric, student reflection, etc., please attach it. 
If you are not attaching a rubric, please explain your criteria for grading the work (if graded). 

Did you share these criteria with students? [ ] yes [ ] no 



6. What criteria did you use to decide which papers are "M" middle papers and which are "H" high? 
(especially if work was not graded originally or if different from #5 above) 



7. Approximately what percent of students performed at the following levels on this assignment: 

% = good-excellent % = adequate % = not yet adequate 
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Cover Sheet for Typical Content Area Writing Assignment (Elementary School Only) 

Please check one: □ science □ social studies □ math 

If you need more room to answer the questions, please use the back of this form or attach sheets as necessary. 

1. Describe the assignment below in detail or attach a copy of the assignment directions to this sheet. 

Be sure to tell us exactly what directions were given to students. 

Specify the title and type (e.g., poem, novel, textbook, etc.) and grade level of any reading material. If 
students are working in reading groups, specify which group was given this assignment. 



2. What concepts, skills, and/or processes did you expect the students to acquire from this assignment? 
i.e., what were your learning goals for the students? 



3. How does the assignment fit in with your unit or what you are teaching in your language arts class this 
month or this year? Is this an end-of-unit assessment? □ yes □ no How long did the task take? 



4. What type of help, if any, did students receive to complete the assignment? (Check all that apply.) 
Students received help or formative feedback from a □ teacher □ teacher's aide □ other students 
□ parents (e.g., help = substantive revision feedback from teacher or peers). Please explain: 



5. How was this assignment assessed? If there is a rubric, student reflection, etc., please attach it. 
If you are not attaching a rubric, please explain your criteria for grading the work (if graded). 

Did you share these criteria with students? [ ] yes [ ] no 



6. What criteria did you use to decide which papers are "M" middle papers and which are "H" high? 
(especially if work was not graded originally or if different from #5 above) 



7. Approximately what percent of students performed at the following levels on this assignment: 

% = good-excellent % = adequate % = not yet adequate 
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Cover Sheet for Very Challenging Assignment or Project: Written Component 

If you need more room to answer the questions, please use the back of this form or attach sheets as necessary. 

1. Describe the assignment below in detail or attach a copy of the assignment directions to this sheet. 

Be sure to tell us exactly what directions were given to students. 

Specify the title and type (e.g., poem, novel, textbook, etc.) and grade level of any reading material. If 
students are working in reading groups, specify which group was given this assignment. 



2. What concepts, skills, and/or processes did you expect the students to acquire from this assignment? 
i.e., what were your learning goals for the students? 



3. How does the assignment fit in with your unit or what you are teaching in your language arts class this 
month or this year? Is this an end-of-unit assessment? □ yes □ no How long did the task take? 



4. What type of help, if any, did students receive to complete the assignment? (Check all that apply.) 
Students received help or formative feedback from a □ teacher □ teacher's aide □ other students 
□ parents (e.g., help = substantive revision feedback from teacher or peers). Please explain: 



5. How was this assignment assessed? If there is a rubric, student reflection, etc., please attach it. 
If you are not attaching a rubric, please explain your criteria for grading the work (if graded). 

Did you share these criteria with students? [ ] yes [ ] no 



6. What criteria did you use to decide which papers are "M" middle papers and which are "H" high? 
(especially if work was not graded originally or if different from #5 above) 



7. Approximately what percent of students performed at the following levels on this assignment: 

% = good-excellent % = adequate % = not yet adequate 
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APPENDIX B 



Classroom Observation Protocol and Observation Interview 
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LAAMP Classroom Pre-Observation Interview 



Researcher 



T eacher F irst Name T eacher Last N ame 



Grade 



Date School Lang Arts Focus 

O Reading O Writing OBoth 

NOTE: If time is limited, go directly to question #3, then ask remaining questions at another time. 

1 . Briefly describe the students in your class, including those with special needs. 



Are there any LEP students in this class? OYes ONo 
If so, approximately what percentage of the students are LEP? 
What languages other than English do students speak? 

2. What should I expect to see during the observation? 



3. What are your goals or objectives for the lesson I will be observing? What skills, concepts, or facts do you want 
the students to learn as a result of this lesson? (e.g., 3rd grade: learning the different parts of a story, learning how 
to peer edit, developing reading strategies such as predicting; 7th grade: learning the structure of a five-paragraph 
essay; increasing vocabulary skills, etc.) 



Pre-Observation Interview p. 1 



4. In what way are these goals suitable for this group of students? 

5. Are these goals based on a specific set of standards? OYes ONo 

If so, which standards (e.g., district standards, state Challenge standards, etc.)? 
How do your goals for this lesson support these standards? 



6. Do you have a formal way of assessing what students have learned in this lesson? (e.g., rubrics, etc.) 



7. Did you plan the lesson we will be observing with other teachers? 



OYes ONo 



Notes 



Pre-Observation Interview p. 2 
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Classroom Information 



Researcher Teacher First Name Teacher Last Name Grade 

Date School Lang Arts Focus 

| | , | |Q Reading O Writing O Both | 




Total no. of minutes observed [[ 



TEACHER AND STUDENT INFORMATION 

No. of students observed [ ~„ | No. of Boys „ No. of Girls | j 



Teacher 

Ethnicity 


2nd Teacher (in a 
team-taught 
classroom) 
Ethnicity 


TA 

Ethnicity 


2nd TA 
Ethnicity 


i i 


1 “ 1 


1 1 


i 


Sex |0 Male O Female | 


Sex |0 Male O Female | 


Sex |0 Male O Female | 


Sex |0 Male O Female | 



Please indicate the number of students who belong to the following ethnic groups. If there is no way to tell, 
please write “missing data” for this section. 



African- Amer. ) | Asian Latino/a 



White | ,| Other 



BILINGUAL CLASSROOM INFORMATIQJYou can mark more than one.) 

1. Number of students given instruction in a second language: | ~| 

2. Language(s) used by the teacher during the observation: |D English □ Spanish □ Other... | 

2a. Percentage of time teacher used English: - \ 

3. Language(s) used by the TA during the observation: |D English □ Spanish □ Other... | 

3a. Percentage of time TA used English: | 

4. Language(s) used by the majority of the students during the obs. : [□ English □ Spanish □ Other. . . | 

4a. Percentage of time students used English: 1 ~| 
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Activity Boxes 
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□ WPITIHO other— Plca5« specify: 

□ Other (fill in) 



Lesson Description 

Please describe the lesson and sequence of learning activities. Remember to include enough 
details so that a person who was not present during the lesson could get a clear picture of 
what the lesson entailed and how it unfolded. Please describe: the different types of 
groupings (e.g., teacher-fronted, small-group with aide, etc.), the use of different types of 
instructional materials in these groups (e.g., computers, books, worksheets), and what 
students were doing in these activities and what for (e.g., participating in a discussion, 
writing in a workbook, etc.). If computer technology was used, please describe: the 
make/model of computers, how many there were in the classroom, how many students were 
using them, what the students were doing (e.g., word processing, graphics programs), and 
purpose for using the computer (e.g., publishing school newspaper, typing out writing 
assignment, etc.). 
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Classroom Ratings 



1. Stated Goals 

In this section rate the clarity and specificity of the teacher’s stated goals for the lesson. Do not rate the 
learning activities. 



Clarity of Stated 
Goals for the 
Lesson 



Ol 

Goals are not clear in 
terms of what students 
are to learn from the 
assignment OR all goals 
may be stated as 
activities with no 
definable objective 
(“activity for activity’s 
sake”). 



02 

Goals are somewhat 
clear and explicit in 
terms of what students 
are to learn from the 
assignment OR goals 
may be a combination of 
goals and activities with 
no definable objective. 



03 

Most goals are clear and 
explicit in terms of what 
students are to learn 
from the assignment OR 
some goals may be 
stated as activities with 
no definable objective. 



0 4 

All the goals are very 
clear and explicit in 
terms of what students 
are to learn from the 
assignment OR all goals 
are elaborated, framed in 
terms of student 
learning. 



Please write a paragraph providing evidence for your rating of the clarity and specificity of the teacher’s 
goals for this lesson. Be as specific as possible and remember to include concrete examples. 
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2. Learning Activities 

In this section rate the degree to which the teacher’s stated goals for the lesson were reflected in the design 
of the learning activities. Specifically rate how well the activity supported achievement of the teacher’s 
goals. Also rate the overall challenge of the learning activities. Additionally, rate the degree to which the 
lesson was effectively implemented, and the degree of student engagement in the activities. 



Coherence 
between Goals and 
Learning Activities 
(e.g. how well the 
activity promoted 
achievement of the 
teacher’s goals) 



Ol 

Learning activities do 
not support or are 
unsuitable to 
instructional goals (e.g. 
activity for activity’s 
sake). 



02 

Learning activities are 
somewhat suitable to 
instructional goals. 



03 

Learning activities are 
mostly suitable to 
instructional goals. 



04 

Learning activities are 
highly relevant to 
instructional goals. 
Additionally, they may 
build on each other to 
provide a coherent 
whole. 



Challenge of 
Learning Activities 



Ol 

Learning activities 
involve students in tasks 
that do not require any 
degree of complex 
thinking. Students may 
be asked to recall only 
very basic information, 
and/or do not engage 
with relevant content 
material. Activities may 
be inappropriate to 
students in terms of age. 
(E.g. a 7th-grade teacher 
reads a story to class and 
asks class to recall very 
basic facts about the 
stories; a 7th-grade 
teacher asks students to 
define very basic 
vocabulary terms.) 



02 

Learning activities 
involve students in tasks 
that require moderately 
complex thinking. 
Students may be asked 
to summarize 
straightforward 
information, infer simple 
main ideas, or 
understand the basic 
format for a given genre 
of writing (e.g. learning 
to apply the format for a 
letter). Limited 
understanding of content 
is required. (E.g. a 7th- 
grade teacher asks 
students to describe the 
beginning, middle, and 
end of a grade-level 
book.) 



03 

At least some of the 
learning activities 
require strongly complex 
thinking as a major 
focus of the lesson, and 
understanding of content 
material is required. 
Students may be asked 
to synthesize ideas, 
analyze cause and effect, 
identify a problem and 
pose reasonable 
solutions, hypothesize, 
speculate giving details 
or justification, defend 
opinions or argue a 
position with evidence, 
evaluate, analyze 
(distinguish important or 
relevant from 
unimportant or 
irrelevant), determine 
bias, values, intent. (E.g. 
a 3rd-grade teacher asks 
students to analyze a 
character from a book 
they read.) 



04 

Much or all of the 
learning activities 
require strongly 
complex thinking as a 
major focus of the 
lesson. Students also 
engage in substantive 
content material. 
Students may be asked 
to analyze cause and 
effect, identify a 
problem and pose 
reasonable solutions, 
speculate giving 
details or justification, 
defend opinions or 
argue a position with 
evidence to a great 
extent. 



Implementation 
of the Learning 
Activities (include 
classroom 
management) 



oi 

The learning activity is 
not effectively 
implemented (e.g. the 
class may be 
disorganized, the teacher 
may lack control). 



02 

The learning activity is 
somewhat effectively 
implemented. 



03 

The learning activity is 
effectively implemented 
(e.g. transitions are 
orderly, teacher has 
control of class). 



04 

The learning activity is 
exceptionally well 
implemented (e.g. 
transitions are 
seamless, almost no 
class time is wasted). 
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Proportion of 


Ol 


02 


03 


04 


Students ‘On- 
Task’ 


Less than half of the 


Approximately half the 


Approximately 85% of 


All students are engaged 


students appear to be on- 
task. 


students appear to be on- 
task. 


the students appear to be 
on-task. 


in the activities. 

Students may also 
initiate or adapt 
activities and projects to 
enhance understanding. 



Please provide evidence for your rating of the coherence and perceived level of challenge of the learning 
activities/lesson. 

Describe the rigor and grade-level appropriateness of the activities and resources used (e.g., did the 
activities or lesson support students’ development of HOT, or meaningful content area knowledge). 

Comment on reasons for why some students and not others may have appeared to be on-task. 

Comment as well on the degree to which students appeared to be interested and engaged in the 
lesson/activities. 

Include concrete examples to support your ratings. 

Provide the titles and authors of the texts used in the classroom. 



O 
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3. Classroom Discussion 

In this section rate students’ opportunity to learn through and engage as partners in meaningful classroom 
discussions. This includes both the nature of the teacher’s questions and the amount of time spent in 
discussion. This also includes the level of student participation. 



Opportunity to 


Ol 


02 


03 


04 


Participate in 


Interaction between 


Teacher makes some 


Most of teacher’s 


Classroom interaction 


Classroom 


teacher and students is 


attempt to engage 


questions are of high 


represents true 


Discussions 


predominantly recitation 
style, with teacher 
mediating all questions 
and answers (i.e., 
teacher’s questions are 
close-ended). OR 
discussion does not take 
place at all (e.g., 
students are working 
individually revising 
drafts, etc.). 


students in true 
discussion, with uneven 
results. Some of the 
teacher’ s questions are 
open-ended. 


quality. Adequate time 
is available for students 
to respond and teacher 
activity solicits student 
input (e.g., “Tell me 
why you think that...,” 
“Can you tell me a little 
more about that...”). 
Teacher builds on 
student contributions. 


discussion. Students 
initiate topics and make 
unsolicited, on-topic 
contributions. Students 
formulate many 
questions. Teacher’s 
questions are uniformly 
high quality, with 
adequate time for 
students to respond. 
Teacher builds on 
students’ contributions, 
and students build on 
each other’s 
contributions. 



Student 


Ol 


02 


03 


0 4 


Participation in 


Teacher involves only a 


Teacher attempts to 


Teacher involves many 


Teacher involves nearly 


the Classroom 


few students in the 


involve all students in 


of the students in the 


all students in the 


Interactions 


discussion. OR no 
discussion takes place. 


the discussion, but with 
only limited success. 


discussion. 


discussion. 



Please provide justification for your ratings. Remember to include all of the following: 

Examples of teacher’s questions, student responses, and techniques the teacher used to include students in 
the discussion. 

Whether certain groups of students were participating or not in the classroom discourse (e.g., were these 
T.F. P students, etc.). 

The degree to which complex language and vocabulary were used by the teacher and modeled for/presented 
to the students AND the extent to which the teacher facilitated student use of more complex language (e.g. 
did the teacher explain unfamiliar words, did the teacher rephrase students’ questions and statements using 
more complex language, etc.). 
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The context and extent of discussion with attention to why discussion may not have been happening (e.g., 
students were reading silently or reading aloud). 



4. Instructional Feedback 

In this section rate the opportunity students have to receive information about their performance and 
progress toward learning goals, and the degree to which this feedback supports learning. Focus on the 
following components of feedback quality: accuracy, substance, specificity, and helpfulness . 



Quality of 


Ol 


02 


03 


04 


Feedback 


Feedback is either not 


Feedback is inconsistent 


Feedback is mostly high 


Feedback is uniformly 




provided or is of 


in quality: Some 


quality (e.g. expectations 


high quality. Provision 




uniformly poor quality. 


elements of high quality 


are made explicit to 


is made for students to 




Feedback may be 


are present; others are 


students; students are 


use feedback in their 




inappropriate (e.g., 


not. Feedback only 


shown examples of good 


learning. Feedback fully 




humiliating, punitive). 


somewhat supports the 


work). Feedback mostly 


supports the attainment 




Feedback does not 
support instructional 
goals. 


instructional goals. 


supports the 
instructional goals. 


of the instructional 
goals. 



Please provide justification for your rating. Remember to provide concrete examples of the type of 
feedback the teacher gave to students. 

Describe how this feedback did or did not support the instructional goals and the assessment criteria (i.e., 
how the criteria the teacher had in mind for knowing whether or not the instructional goals had been met by 
the students). 
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APPENDIX C 



Language Arts Assignments 

In the following sections we describe each dimension, and then we present 
examples of practice (basic and high quality) for each level of schooling in order to 
clearly illustrate our framework for describing the quality of students' learning 
environments. Specifically we describe the cognitive challenge of the assignments, 
clarity of the goals for student learning, clarity of the grading criteria, alignment of 
learning goals and task, alignment of learning goals and grading criteria, and the 
overall quality of the assignment. We also present the frequencies for each of the 
scale points for the different assignments collected at both levels of schooling in 
Tables Cl to C4. 

Cognitive Challenge of the Assignments 

The first dimension of quality of classroom assignments describes the level of 
thinking required of students to complete the assignment tasks. Specifically, the 
degree to which students have the opportunity to think critically, predict, analyze, 
and synthesize information, as well as engage with substantive content material is 
examined. Teachers' assignments for this sample of schools typically required 
students to use a limited degree of moderately complex thinking. For example, 
students were typically asked to summarize straightforward information, infer a 
simple main idea, or simply apply an appropriate format for a given genre without 
substantive engagement with content material (e.g., write a business letter to a 
movie star or an essay listing suggestions for improving the schools, etc.). 

The following assignment was considered to be of basic quality for the level of 
cognitive challenge observed in classrooms and is typical of what we collected from 
our sample of urban third-grade classrooms. For this writing assignment, students 
were asked to write a story with a beginning, middle, and end using the following 
new vocabulary words: abominable, snow-covered forest, and snowman. Students 
were provided a story starter: Oh no! I was lost! I turned and saw. . . This assignment 
was scored a 2 for the level of cognitive challenge since students were not asked to 
employ the steps of the writing process and were not asked to elaborate, provide 
details, or provide more than one event description. In short, this writing 
assignment seemed to essentially be a version of the standard vocabulary 
assignment of writing new vocabulary words in sentences. The following is a 
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sample of student work considered by the teacher to be of "average" quality for the 
class 1 : 



The Abominable Snowman 

Oh no! I was lost! I turned and saw an Abominable snowman. I ran as fast as I 
could to my mother's cabin. The snowman was big and scary. He had legs that 
made him run very fast. He had a stick in his hand. I was running then he tripped 
me with the stick and I fell. This story happend in snowfall forest and suddently the 
sun came up. It melted him and I was saved by the sun. 



For a reading comprehension assignment similarly scored for the level of 
cognitive challenge in seventh-grade classrooms, students read a few short stories 
and were asked to answer questions about these stories at the end of each section. 
The following are examples of the types of questions asked of students for this 
assignment. 

Why didn't the gods want humans to have fire? How did Prometheus plan to steal fire? 

What happened to Prometheus after he stole the fire? 

This assignment was considered to be of basic quality for this dimension 
because most of the questions asked students to recall only basic facts. Only one 
question required students to use complex thinking and give a justification for their 
response: "Which theory do you think best explains the formation of the moon? 
Why?" The following is an example of student work considered by the teacher to be 
average for the class. 



Seventh-Grade Student Work Sample 

1) Why didn't the gods want humans to have fire? 

They did no want humans to have fire because the though that the humans would have more 
power than them and be powerful 

2) How did Prometheus plan to steal fire? 

He plan by making them believe he left and hiding in the rocks and waiting for them to go to 
sleep and them he would steal the fire. 

3) What happened to Prometheus after he stole the fire? 

The gods punished Prometheus by changing him to a rock on the mountain. 



1 



All student work samples show actual student responses. Original errors have been retained. 
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Assignments considered to be of high quality for this dimension, in contrast, 
required strongly complex thinking as an extensive, major focus of the task and also 
required students to engage with substantive content material. Students may have 
been asked to synthesize ideas, analyze cause and effect, identify a problem and 
pose reasonable solutions, hypothesize, speculate with details or justification, 
defend opinions or argue a position with evidence, evaluate, analyze by 
distinguishing between the important or relevant from the unimportant or 
irrelevant, and determine bias, values, and intent. 

The following assignment, which we considered to be cognitively demanding, 
required students to analyze quotes taken from the book By The Great Horn Spoon 
(Fleischman, 1963), which the teacher and students had read together as part of a 
unit learning about the Gold Rush. Students responded in writing after participating 
in small group discussions. One of the tasks in which students were asked to explain 
quotes from the book is this: 

On p. 97, Praiseworthy made a list of the names of the gold mining camps, such as 
Grizzly Flats, Bedbug, Whiskey Flat and Hangtown. He says, "They sound like dreadful 
places to take a growing boy." But then the author writes "they sounded glorious to 
Jack." 

What does Praiseworthy mean? What does Jack mean? 

This assignment was rated high for cognitive challenge because it required 
students to think deeply about the story and the characters. The book is on a fourth- 
grade level and was used in a combination third- and fourth-grade class. Students 
had to interpret the quotes in light of what they understood about the characters, the 
period of history, and the plot. It is important to note, however, that students were 
not always able to successfully interpret the quotes, thus underscoring an earlier 
point we made that quality of assignments is a necessary but insufficient condition 
for ensuring quality student work. This is illustrated in the following sample of 
student work considered by the teacher to be of average quality for this class. 

Praiseworthy means that jack cant go there because they are like dedfuls places. And he is to yung 
to go in does places, but they have to chose a place to go and I think that they choes the town. I 
think that Jack says that he is supris about that. But I think that he is scared about that and I say 
he dosent want to go there because he thinks that is dedful to. 
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The following assignment, also given a high score for the level of cognitive 
challenge, required seventh-grade students to create a newspaper based on their 
readings of American war novels. The newspaper had to include several types of 
writing: a "cause and effect" essay, a novelty article (such as a dance advertisement 
or a recruitment poster), and two of the following: a first-hand biography, an 
observation, or an evaluation. This assignment was rated high for level of cognitive 
demand because it required that students bring together and analyze different types 
of information as demonstrated in their cause-and-effect essays and evaluation. In 
addition to engaging with substantive content area knowledge (history), this 
assignment is also exceptional in that it required students to write in different styles 
(creatively, analytically, biographically, etc.), thus giving them experience writing in 
(and differentiating between) different genres. An excerpt of student work 
considered by the teacher to be average for the class is shown below. 



December 9 5 cents Honolulu, Hawaii 

The Corps Correspondence 



The Japanese Bomb Pearl Harbor 

The Japanese Armed Services attacked the 
United States Island Naval base at Pearl Harbor, 
Honolulu, Hawaii. The Japanese flew in and 
attached the famed battleship row. The attack 
destroyed many of the battleships along with 
many other American ships of the Pacific fleet. 

An Insiders First Hand Account of the Attack 
of Pearl Harbor As Seen By Lt. Kenneth 
McCoy 

Japanese bombs littered the sky above un 
unsuspecting Pearl Harbor on December 7 th , 
1941. When I arrived on the Island most of the 
attack was already over, but there were still 
fires burning and ammo lockers exploding in 
the ships and on shore. 



As the sea plane I was a passenger on came in 
for its final approach, we were fired upon by a 
group of nervous American anti-aircraft 
batteries. When I stepped out of the plane I saw 
a normally calm vacation site now was a place 
full of death and destruction. I saw the majestic 
skeletons of the once powerful pacific fleet. All 
around was a feeling of remorse and fear. It was 
apparent that a few moments can change the 
course of history 
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Clarity of Assessment Criteria 

In addition to collecting their assignments, we also asked teachers to provide 
us with a copy of their grading criteria (e.g., their scoring rubric) for assessing their 
students' work. We then examined the quality of the teachers' grading criteria in 
terms of their specificity and potential for helping students improve their 
performance. How clearly the teacher defined each aspect of the grading criteria was 
considered, as well as the amount of detail provided for each dimension. 

Overall we found that teachers' criteria for grading student work tended to be 
only moderately clear and explicit. Teachers frequently used a very general or 
rudimentary rubric to score student work or provided us with a list of scoring 
dimensions such as "style, creativity, and organization." For example, in one third- 
grade assignment considered to be of basic quality for this dimension, students were 
required to write a retelling of the story "My Great Aunt Arizona" after listening to 
it read aloud . 2 The teacher's grading criteria for this assignment were quite broadly 
stated. In her words: 

Did they understand what was read? Can they write a beginning, middle, and end? 

The clarity and specificity of her criteria were considered fairly basic because it 
was not clear what a student needed to do to show that he/ she understood what 
was read (for example, how much detail the teacher expected students to include in 
their stories). The teacher also did not specify her criteria for what constituted a 
relatively high- or low-quality part of a story (i.e., beginning, middle, and ending). 

Assignments that received a high score for this dimension, in contrast, had 
assessment criteria that were very clear and detailed. Additionally the teacher may 
have shared the criteria with students ahead of time and/or provided students with 
a model of excellent work. For example, this third-grade assignment considered to 
be of excellent quality for the clarity of the assessment criteria required students to 
write a story using the "writing process including drafting, editing, revising, and 
publishing." To assess the stories the teacher used a rubric which was both clear and 
elaborated. The teacher's rubric for scoring this assignment is shown below. 



2 This assignment was described as well in Aschbacher (1999). 
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1 


2 


3 


4 


does not use 
descriptive words or 
details 


uses few descriptive 
words and/ or details 


uses some descriptive 
words and details 


uses many descriptive 
words, details and 
transition words 


no paragraph structure 


little knowledge of 
paragraph form 


some understanding of 
paragraph form 


uses correct paragraph 
form (indenting, topic, 
supporting and 
concluding sentences) 


one sentence 


contains few sentences 


contains minimum of 5 
sentences 


contains more than 5 
sentences 


unrelated sentences or 
not on topic 


some sentences not 
about topic or 
sentences are 
randomly sequenced 


writing stays on topic 
most of the 
time — some sentences 
out of sequence 


writing stays on topic 
and follows logical 
sequence 


no sentence structure 


repeated or simple 
sentence structure 


all sentences are 
complete with some 
variation in length 


all sentences are 
complete and of 
varying lengths 


frequent use of "and" 


inappropriate uses of 
"and" 


some evidence of 
sentence combining 
and use of commas in 
a series 


evidence of sentence 
combining and use of 
commas in a series 


primarily inventive 
spelling — random 
letters or numbers 


predominantly 
appropriate inventive 
spelling — some high 
frequency words 
spelled correctly 


most high-frequency 
words spelled 
correctly. Evidence of 
ability to use various 
resources to aid 
spelling. 


correct spelling used 
most of the time (high 
frequency all correct) 
uses a variety of 
resources to spell 
difficult words 
correctly 


little correct 
punctuation, 
capitalization, or 
grammar 


some correct 
punctuation, 
capitalization, and 
grammar 


correct punctuation, 
capitalization, and 
grammar used most of 
the time 


correct punctuation, 
capitalization, and 
grammar used almost 
all of the time 


no spacing between 
words and sentences 


some errors in spacing 
between words and 
sentences 


correct spacing 
between all words and 
sentences 


correct spacing 
between all words and 
sentences 


illegible or hard to 
read 


parts of writing are 
illegible or difficult to 
read 


neatly written and 
easy to read 


neatly written and 
easy to read 



At the seventh-grade level, the following assignment was also scored a 4 for the 
clarity of the assessment criteria. For this assignment, students were required to 
write a review of a book of their choice following a detailed outline provided by the 
teacher. This assignment received a high score for the clarity of the teacher's grading 
criteria because students were given a very detailed list of what they needed to 
include in their papers and the criteria upon which their work would be assessed. 
Additionally, the teacher provided a model of excellent work for the students. The 
teacher's outline for this assignment is shown below. 
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I. Introduction 

A. Opening statement: Try to describe the book as best you can in one clear 
sentence. Include the title and author, (example: Pacific Crossing, by Gary Soto, 
is the story of one boy's experience during a summer spent in Japan. 

B. Explain the setting and give a little more detail on the story. 

C. Mention three important episodes from this story that you will talk about in 
the next three paragraphs. 

II. First episode 

A. Explain the beginning of the episode and give any background info needed to 
understand the episode. 

B. Summarize what happens and how the episode turns out. 

C. Tell what your main character learned from the events you describe. 

III. Second episode 

Follow instructions for paragraph II. 

IV. Third episode 

Follow instructions for paragraph II. 

V. Conclusion 

A. Say something about the book overall that connects this paragraph with the 
three that came before it. 

B. Explain what you think the important meaning of the book was, or what is 
valuable to leam from your book. 

C. Make a recommendation and give at least two reasons to support your 
recommendation. You can make a negative recommendation if you wish, just 
be sure you have two good reasons to support it. ("Dumb" or "boring" are not 
good reasons — you need to say why it is dumb or boring.) 

Alignment of Goals and Grading Criteria 

We also examined the degree to which teachers assessed students on the 
explicit skills and concepts mentioned as learning goals for the assignment. 
Typically we found that teachers' assignments demonstrated only some degree of 
alignment between the teacher's stated learning goals and the grading criteria; or, 
the teacher's goals and assessment criteria were so vague that alignment occurred 
only at a very general level. For example, in one third-grade writing assignment 
considered to be of basic quality for this dimension, students were asked to write a 
report about a topic of their choice. The teacher's learning goals for this assignment 
were "Brainstorming, drafting, learning to write report of information, and putting 
meaning to text." The teacher then assessed students' work according to "content 
and written expression." The alignment between the task and learning goals was 
considered to be somewhat low since the assessment criteria were vaguely stated 
and seemed to reflect only part of the goals . 3 



3 This assignment also was described in Aschbacher (1999). 



To receive the highest score for this dimension, an assignment would need to 
have goals and assessment criteria that overlapped completely. In other words, the 
teacher would have had to clearly consider and plan for a way to assess the degree 
to which students had achieved each of the assignment's learning goals. This is 
illustrated in the following seventh-grade writing assignment scored a 4 for this 
dimension. Here the teacher asked students to "produce a research paper on an 
inventor, artist, writer, composer, scientist, or musician in which they focus on 
his/her creative accomplishments, identify the person's most creative works, 
summarize the person's life history, and show how they reflected the times in which 
they lived." Specifically, students were to address the following questions: 

1. What is creativity? 

2. In what ways are people creative? 

3. Why are the arts important to human life? 

4. In what ways do inventors, scientists and artists reflect the time in which they live? 

The teacher's stated learning goals for the assignment were the following: 

I WANT MY STUDENTS TO KNOW . . . 

1. Creativity comes in many forms 

2. Students are capable of creativity and creative responses 

3. Creativity is not only in product but in process as well 

4. The difference between inventors and discoverers (scientists and doctors) 

5. How to do formal research 

I WANT MY STUDENTS TO BE ABLE TO... 

1 . Outline prior to writing 

2. Use resources effectively, i.e., encyclopedia, newspapers, books, magazines, CD 
encyclopedias to gather pertinent information 

3. Use quotation marks correctly and paraphrase correctly 

4. Summarize gathered information 
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5. Write a research report with a clear focus 



6 . Organize writing coherently 

7 . Elaborate statements with facts, examples , details 

8. Word process 

9. Revise for clarity 

An excerpt from the teacher's assessment criteria is provided below. 

An "A " paper had to present or contain : 

1 . An overview of the person's life , including the important moments 

2. Important achievements 

3 . What prompted the person to do what s/he did 

4 . How the poem reflected his/her times 

5. A conclusion which stated the writer's feeling about the subject of the report 

6 . A bibliography 

This assignment was given a high score for this dimension because both the 
goals and the grading criteria were clearly stated and elaborated. Furthermore, there 
was a direct correspondence between what the teacher wanted students to know 
and be able to do and what the teacher looked for when scoring the papers. For 
example, the goal for students to "write a research report with a clear focus" 
corresponds to the point in the rubric that says, "focused on topic with a clear thesis 
aimed at describing the person's life, creative accomplishments, and why the person 
was important." 

Tables Cl to C4 illustrate the frequencies for each of the scale points for the 
different assignments collected at both levels of schooling. 



O 

ERiC 



53 59 



Table Cl 

Scale Percentages for Reading Comprehension Assignments in Elementary and Middle School 



(N = 24 Teachers) 




Elementary schools 


Middle schools 


1 


2 


3 4 


1 


2 


3 4 


Level of cognitive challenge 


25.0 


58.3 


16.7 — 


27.3 


54.5 


18.2 — 


Clarity of learning goals 


8.3 


75.0 


16.7 — 


18.2 


81.8 


— — 


Clarity of grading criteria 


27.3 


54.5 


9.1 9.1 


45.5 


54.5 


— — 


Alignment of goals and task 


16.7 


66.7 


16.7 — 


9.1 


72.7 


18.2 — 


Alignment of goals and grading criteria 


45.5 


45.5 


9.1 — 


54.5 


45.5 


— — 


Overall quality 


25.0 


58.3 


16.7 — 


27.3 


72.7 


— — 



Table C2 

Scale Percentages for Writing Assignments in Elementary and Middle School (N = 24 Teachers) 



Elementary schools Middle schools 





1 


2 


3 


4 


1 


2 


3 


4 


Level of cognitive challenge 


16.7 


75.0 


8.3 


— 


8.3 


66.7 


25.0 


— 


Clarity of learning goals 


8.3 


83.3 


8.3 


— 


17.4 


69.6 


13.0 


— 


Clarity of grading criteria 


33.3 


41.7 


16.7 


8.3 


30.4 


47.8 


13.0 


8.7 


Alignment of goals and task 


16.7 


58.3 


25.0 


— 


8.7 


69.6 


21.7 


— 


Alignment of goals and grading criteria 


16.7 


75.0 


8.3 


— 


40.9 


40.9 


18.2 


— 


Overall quality 


25.0 


66.7 


8.3 


— 


16.7 


62.5 


20.8 


— 



Table C3 

Scale Percentages for Content Area Writing Assignments in Elementary 
School (N = 12 Teachers) 



Score point 



12 3 4 



Level of cognitive challenge 


8.3 


91.7 


— 


— 


Clarity of learning goals 


41.7 


58.3 


— 


— 


Clarity of grading criteria 


27.3 


63.6 


9.1 


— 


Alignment of goals and task 


16.7 


75.0 


8.3 


— 


Alignment of goals and grading criteria 


54.5 


36.4 


9.1 


— 


Overall quality 


25.0 


75.0 


— 


— 
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Table C4 

Scale Percentages for Challenging Major Project Assignments in Elementary and Middle School 
(N = 24 Teachers) 





Elementary schools 




Middle schools 


1 


2 


3 4 


1 


2 


3 4 


Level of cognitive challenge 


— 


58.3 


41.7 — 


9.1 


45.5 


45.5 — 


Clarity of learning goals 


— 


83.3 


16.7 — 


27.3 


54.5 


18.2 — 


Clarity of grading criteria 


18.2 


45.5 


18.2 18.2 


10.0 


60.0 


30.0 — 


Alignment of goals and task 


— 


58.3 


41.7 — 


18.2 


63.6 


18.2 — 


Alignment of goals and grading criteria 


27.3 


54.5 


18.2 — 


20.0 


80.0 


— — 


Overall quality 


— 


75.0 


25.0 — 


9.1 


54.5 


36.4 — 





APPENDIX D: Rubric for Scoring Teachers' Language Arts Assignments 
Language Arts Assignment Rubric - Draft 
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APPENDIX E: Rubric for Scoring Students' Writing Assignments 
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APPENDIX F 

Examples of Instructional Practices From Classroom Observations 



In this section we report data from field notes to 'illustrate our ratings for key 
dimensions of quality observed in our sample elementary and middle schools. First 
we describe the dimension, and then we present two examples of practice (basic and 
high quality) for that aspect of quality practice. We pay special attention to the level 
of cognitive challenge of the lesson activities, instructional conversation (classroom 
discussions), instructional feedback to students, the alignment of learning goals and 
lesson activities, and lesson implementation (classroom management). The purpose 
of this is to illustrate the range of observed practice in these schools as well as to 
illustrate clearly our framework for describing the quality of students' learning 
environments. 

Table FI presents the mean ratings for the different dimensions of quality 
observed in our sample elementary and middle school classrooms (1 = poor, 
4 = excellent; N = 26 teachers). Commensurate with the quality of assignments, we 
found that the level of cognitive challenge of the lesson activities we observed 
tended to be somewhat low overall (M = 2.19 and 1.92 in elementary and middle 
schools, respectively). This was true as well for the quality of instructional 
conversations in classrooms (M = 1.54). In contrast, student engagement and the 
quality of lesson implementation (classroom management skills) tended to be better 
on the whole, notably in the elementary school classrooms (M = 2.96). 



Table FI 

Quality of Observed Language Arts Lessons (N = 26 Teachers) 





Elementary school 
lessons (N = 26) 
M(SD) 


Middle school 
lessons (N = 26) 
M (SD) 


Challenge of lesson activities 


2.19 (0.75) 


1.92 (0.80) 


Quality of classroom discussions 


1.54 (0.65) 


1.54(0.71) 


Level of student participation in classroom discussions 


1.85 (1.01) 


1.69 (0.84) 


Quality of instructional feedback to students 


1.85 (0.83) 


1.96 (0.82) 


Level of student engagement in the lesson 


3.00 (0.63) 


2.62 (0.75) 


Quality of lesson implementation (classroom management) 


2.96 (0.87) 


2.27(1.08) 


Clarity of the teachers' goals for the lesson 


2.65 (0.80) 


2.19 (1.02) 


Alignment of learning goals and lesson activities 


2.42 (0.70) 


2.50 (0.95) 



Note. Items were scored on a 4-point scale (1 = poor, 4 = excellent). 
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Cognitive Challenge of the Lesson Activities 

An important part of our classroom observations was looking at the level of 
thinking required of students to participate in the lesson activities. For example, we 
documented the degree to which students had the opportunity to think critically, 
predict, analyze, and synthesize information, as well as engage with substantive 
content material. 

On average, we found that the level of cognitive challenge of the lesson 
activities observed in classrooms was fairly basic. For example, the type of thinking 
typically required of students in the third-grade classrooms generally included 
answering low-level comprehension questions in reading groups (e.g., "What color 
was the frog?") and completing worksheets. In the seventh-grade classrooms we 
observed, the type of thinking typically required of students included summarizing 
straightforward information, inferring simple main ideas, and understanding the 
basic format for a given genre of writing (e.g., learning to apply the format for a 
letter). 

The following illustrates a typical language arts lesson we observed with 
regard to the cognitive challenge dimension. In this seventh-grade classroom the 
teacher gave students a Venn diagram worksheet with three circles titled "Character 
Comparison." Students were asked to describe two characters in the novel using 
physical traits, some basic personality traits, and behavior. This lesson was 
considered to require some degree of higher order thinking skills since the activity 
and supporting discussion for the most part centered on recalling descriptive 
characteristics about the characters most of which were explicit in the text. This is 
illustrated in the following exchange between the teacher and the students: 

T: Let's describe Omri. Is she stock y or lanky? 

Students give different answers. 

T: Let's go to the book. On the bottom of page 3, we learn that Omri has blonde hair and Ellen 
has dark hair. How old were each of these girls? 

S: Ten. 

T: They were friends and were 10 years old. I think if I remember correctly that Ellen was stocky 
and Omri was lanky. So you could put that down for their physical traits. What else could 
you say about Omri? 

S: She had a sister. 
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T: You will have far more to put down in these circles than you will have lines for. What could 
you say about their personality? What did Omri exhibit? 

S: Courage. 

T: What was Ellen in this story? 

S: Afraid. 

T : Yes, she was afraid. Was she brave? -When did she exhibit bravery? 

The following illustrates two lessons from the third and seventh grades that we 
considered to be of high quality with regard to the level of cognitive challenge. In 
this third-grade lesson, the teacher led guided reading groups while the rest of the 
students wrote in their writing journals or read silently. The teacher skillfully led 
each reading group, guiding students to use effective reading strategies such as 
making predictions and using contextual clues to help extract meaning from the text. 
For example, when the students in one group were first given their books, she 
suggested they go ahead and read the summary of the book on the back cover and 
the chapter titles and look at the pictures for information about the characters and 
the plot. As illustrated in the following example, the teacher pushed and probed to 
get students to learn as much as they could before beginning to read the first 
chapter: 

T: When you get a new book, what is the first thing you want to do? 

S: Read the back. 

T: Read the back and see what it is about and see if you think you will like it. 

T: Who do you think the characters might be? 

S: Mom. 

T: Where does it take place? What is the setting? 

Students answer various things. 

T: Who is Amber Brown? 

T: How old do you think she is? What kind of a girl do you think she is? What grade do you 
think she is in? 

Students answer these questions. 

T: What are you basing that on ? Does it say somewhere? 
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T: Do you think she is in first grade? 

S: She looks like she is older. 

T: What do you think the title means? 

S: She wants advice. 

T: What do you think the plot is? The main idea? 

S: Amber Brown wants extra credit because she does all the work. 

T: Okay. Look at the very first sentence of the summary. "Amber Brown is in deep trouble." 

What do you think the problem of the story will be about? 

S: She's fighting with a boy. 

T: Based on what you read , not a wild guess. 

S: She is in trouble. 

T: Do you think she will get out of trouble? 

T: Your job this morning is to read chapters one and two. You have until tomorrow to read also, 
with a partner or silently. 

This lesson was rated high, in part because the teacher required the students to 
use available evidence for predicting the future events in the story and actively 
engaged students in a discussion based on this evidence. Additionally, the teacher 
modeled for the students how to read for meaning and presented them with various 
reading comprehension strategies for doing so (e.g., looking at the cover, reading the 
back of the book, reflecting on the title). Other students who were not part of the 
reading group read silently or with a partner or wrote in their journals. All of the 
students appeared to be engaged in these activities. 

The following is an example of a seventh-grade language arts classroom that 
received a high score for the level of cognitive challenge. The class began with 
students doing a "quick-write" predicting what would happen in the next few 
chapters of the novel they were currently reading about the Salem witch trials. This 
exercise was done to prepare them to meet in small groups to discuss their book. 
The teacher then led the class in a discussion based on the themes of freedom and 
courage. In this discussion the teacher made references back to other books the class 
had read. This is illustrated in the following exchange: 
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S: It seems like the Dark Ages. 

T: Why is that? 

S: Because everyone was scared and uncertain about things during the Dark Ages. 

T: Does anyone recall what books we have read where freedom and fear were underlying themes ? 

Ss: Pride and Prejudice. 

After a substantial discussion, the teacher's aide handed out an article from The 
Los Angeles Times on gangs. The teacher spoke with the students about the concept of 
persecution and asked students to compare the persecution of gang members with 
the persecution of a character in the novel they are reading based on the Salem witch 
trials. Students then broke into groups to discuss their novel. This language arts 
class was considered to be of high quality for the level of cognitive challenge 
because there was a sophisticated discussion of the themes of personal choice, 
persecution, etc. that went beyond one book and time period (Salem witch trials) to 
connect across historical periods and genres. 

The Quality of Classroom Discussions 

In observing classrooms we also examined the extent to which students had the 
opportunity to learn through and engage as partners in meaningful classroom 
discussions during the lesson activities. We specifically focused on the nature of 
teachers' questions (e.g., degree of open-endedness) and the degree to which 
teachers built on and extended student contributions. 

This dimension of practice tended to be the weakest overall, and we do not 
have an example of a high-quality discussion to report for this section. In general we 
found that while teachers often solicited student contribution during the lesson 
activities (e.g., asked a few open-ended questions), this did not often turn into a 
high-quality classroom discussion. This is illustrated in one third-grade classroom 
where the teacher asked some open-ended questions during the lesson but was 
largely unsuccessful in getting students to respond. Furthermore, when the students 
did respond, the teacher would frequently not follow up on their contributions. The 
following is an example of the classroom discussion during the observed lesson. 

T: I want us to do a word wall for this book. [The teacher sends a child to get a marker and 
poster.] Who wants to write it? Okay, Juan, you write it. On a word wall, you don't have to 
just write words. For example, I like this part. My favorite phrase besides "dusk" is "trees 
glittering." Look through your reading and pick out a part you really like. 



ERIC 



62 



70 



S: "Drifted." 

T: How does it make you feel? What does it make you think? 

No answer. 

T: Can’t think of it? Look through your books and find words you really like. 

The students hesitate and then begin opening their books. 

T: I'm going to add my words. [She takes the poster and begins writing " trees glittering."] I 
don't know how to write it-g, l, i, t. [She tries to get help from the students, but gets no 

response.] 

S: [Reading from what he wrote on the wall.] "Snow drifted through the streets." 

T: What does snow drifting through the streets look like to you? 

S: Like white snow blowing in the wind. 

S: [A different student reads his phrase.] "Steaming tamales. " 

S: [A different student reads his phrase.] "Cheerfully. " 

T: What does cheerfully look like? 

No answer. 

Quality of Instructional Feedback 

In addition to looking at students' opportunity to participate in high-quality 
instructional conversations, we also examined their opportunity to receive 
information about their performance and progress toward learning goals. 
Specifically, we focused on the accuracy, substance, specificity, and helpfulness of 
the teacher's feedback to students, as well as the amount of feedback the teacher 
provided to students during the observed lesson activities. 

We found that, in general, teachers' feedback to the students tended to be 
inconsistent in quality. By this we mean that students either did not receive much 
instructional feedback during the lesson, or the quality of the feedback they received 
was limited. This latter situation is illustrated in the following lesson. For a large 
portion of the observed class period the teacher met with students to review 
sentences they had written. During this time the teacher showed students the 
sentences they had written that contained mistakes. She did not often provide them 
with guidance for understanding why they had made the mistake, however, and 



what they needed to do to improve their writing generally. The following are 
examples of the exchanges between this teacher and her students: 

A student comes up to have her sentences checked. 

T: Okay, " I was cautiously" what? Okay, and this doesn't make sense either, "I was cautiously 
and careful." Go back and fix those. 

A student shows his work to the teacher. 

T: "My dad was dignified in his business field in a proper way"? Can you fix that please. 

A student shows her sentences to the teacher. 

T: "I was cautiously and careful with my mom's dishes . " That doesn't make sense. Go fix 2 and 4. 

We rated lessons high on this dimension if teachers not only provided students 
with correct answers but also modeled strategies to students to figure out answers. 
We also looked to see whether the teacher provided consistent, detailed, and specific 
information to students about their performance, and whether students had many 
opportunities to show what they knew. 

In one third-grade class rated high on this dimension, the teacher's feedback to 
the students was consistently of high quality and appeared to help the students 
learn and progress toward the goals she had set for them. The teacher constantly 
assessed students' understanding and used the information to modify instruction, 
give feedback, and help individual students. This is illustrated in the following 
third-grade lesson. 

While the class read the morning message aloud, some students read night for 
the written word evening. The teacher used that as an opportunity to have the 
students look carefully at the first letter and asked the class, "If the word was night, 
what would be the first letter?" Most of the teacher's questions and comments also 
seemed designed to steer the students to read for meaning. She suggested reading 
strategies to a student who misread an important word, not only so he would read 
the word correctly, but so he could learn an effective reading strategy. For example, 
when the student came to a word he did not recognize — surrounded — the teacher 
asked, "Do you see part of a word you know?" The student immediately said the 
word correctly. 

During this seventh-grade lesson rated high for the quality of instructional 
feedback, the teacher gave students many chances to share their writing with the 
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class and took advantage of this to give students plenty of focused feedback. The 
following is one example: 

T: Raise your hands if you have a good metaphor you'd like to share with the class. 

S: Summer camp is a vacation away from my family. 

T: [Writes it on the overhead.] She is saying summer camp is a vacation. Now, these two are 
similar, so it's kind of stretching the idea that this is a metaphor. But we can still work with 
it. Now, she definitely has the idea that a metaphor uses is_a and not js like a, so very good, 

Lisa. 

Later in the class period the teacher provided the students with examples of 
good and bad writing and solicited from them their ideas about what made the 
writing good or bad and how to improve the writing. During this time the teacher 
was careful to follow up a student's suggestion with feedback about what was good 
and bad about a specific example. A few times, the teacher only said, "Very good, 
thank you," after a student's suggestion. For the most part, however, she pointed 
out something specifically right or wrong with students' suggestions. 

Alignment Between Goals and the Lesson Activities 

We also considered the degree to which the lesson activities appeared to 
further the teachers' goals for student learning. We found that teachers' stated 
learning goals and lesson activities were typically only somewhat aligned in most 
classrooms. This is illustrated in the following seventh-grade lesson considered 
typical for this dimension. Here the teacher's stated goal for the lesson was that the 
students learn how to use a Venn diagram to help them organize their thoughts as 
part of the prewriting process. The class spent only 14 minutes on the Venn diagram 
activity, however. The rest of the 50-minute class period was spent having students 
fill out their reading logs, review words from the vocabulary cards, and read aloud 
the beginning of a story. The teacher did not mention these other activities when 
stating her goals for the class period, so it is not clear how she intended these other 
activities to further student learning. For this reason we considered this lesson to be 
somewhat low for this aspect of quality practice. 

The lessons that rated high on this dimension had goals and activities that 
overlapped completely. Every activity had a goal for student learning attached to it, 
and the activities appeared to further the attainment of the goal. For example, every 
one of this third-grade teacher's goals was addressed during one observed lesson. 
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Furthermore, the activities all appeared to effectively further the teacher's goals — for 
example, journal writing to get students to write independently, morning message 
to teach phonics, reading lesson to teach story structure, and Writer's Workshop 
(including Author's Chair) to teach the writing process. Additionally, all of these 
activities were tied together. For example, the teacher used student-created 
"Excitement Graphs" to show that every story builds up to a climax and then asked 
the students to apply this in their own writing during the Writer's Workshop. 

Similarly, all of this seventh-grade teacher's goals for student learning were 
addressed during the observed lesson, and every activity had a learning goal 
attached to it. For example, the teacher had the students read an article based on 
their own interest to learn to read for pleasure, write on a personal topic to learn 
how to be more descriptive, become familiar with good reading habits through 
reviewing a text together on that subject, and improve their listening skills by 
hearing her read a story out loud to them. Additionally, most of the teacher's goals 
were geared toward improving reading comprehension; thus smaller goals and 
individual activities together appeared to build on each other during the observed 
lesson to create a coherent whole. 

Lesson Implementation/Classroom Management 

In addition to the challenge and design of the observed lesson activities and the 
quality of the verbal interactions between the teacher and students, we also 
examined teachers' classroom management skills during our observations. 
Specifically, we noted how difficult or off-task behavior on the part of students was 
handled (e.g., how effectively student behavior was dealt with, how positive the 
teacher was toward the students, etc.) and the amount of time spent transitioning 
from different activities. 

In the elementary school classrooms, the quality of classroom management 
tended to be good overall. Classrooms were generally well organized, and 
considerably less time was spent in transitions from activity to activity in 
comparison with the seventh-grade classrooms. An example of the type of 
management skills we typically observed in third-grade classrooms is illustrated in 
the following third-grade lesson. Here the teacher set up the activities and the class 
to make it possible for students to continuously work while she met with three 
different guided reading groups. It took some time for the teacher to get organized 
(e.g., remembering how many pages the students were to have read) and begin each 
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group. For the most part, however, the teacher was positive toward the students, the 
class ran smoothly, and there was little wasted learning time. 

In contrast to the elementary schools, the classrooms tended to be less well 
managed in the middle schools. This is perhaps due in part to the fact that the 
classes were much larger in middle schools than in elementary schools (an average 
of 32 versus 20 students respectively). Additionally, there were more difficult or 
disruptive behaviors on the part of students, which were not always effectively 
handled by teachers. For example, in one typical seventh-grade classroom the 
teacher continually kept saying "Shh!" to the students, though this did little to quiet 
the room. For the most part, this teacher ignored the students' talking and 
disruptive behavior. 

In classrooms given the highest rating for this dimension, the transitions 
between activities were seamless and the teacher handled off-task or disruptive 
behavior in an effective and positive manner. For example, in one third-grade 
classroom rated a 4 for the quality of the lesson implementation, the students talked 
to each other during parts of the lesson, but most of this talk was on task and 
appropriate. When these discussions were not on task, the teacher handled it in a 
very nice way, not embarrassing anyone. This was illustrated during "Shared 
Reading" time when the teacher asked the class what they liked about the book the 
teacher was reading. Two boys were talking inappropriately when a soft-spoken girl 
answered the question. The teacher nicely said, "Boys, I can't hear Sherry." The boys 
stopped talking, and everyone listened as Sherry gave her answer. 

In a seventh-grade classroom also given the highest rating for this dimension, 
the teacher was respectful to students and appeared to be respected in return. For 
example, all of the students stayed quietly in their seats after the bell rang, waiting 
to be dismissed. Additionally, little instructional time was wasted. For example, the 
teacher had students work on a writing task while she took 5 minutes to take roll 
instead of wasting learning time on a (necessary) procedural task. The transitions 
from whole class to small groups were also very efficient. The teacher also gave very 
clear directions to the students in a nice way, and most students appeared to know 
what to do and what was expected of them during the lesson. 
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APPENDIX G 



Intercorrelation of Ratings for Individual Assignment Types 



Table Gl 

Interrcorrelation of Ratings for the Reading Comprehension Assignments (N = 36) 





Cognitive 

challenge 


Clarity of 
goals 


Grading 

criteria 


Alignment 
goals/ task 


Alignment 

goals /grade Overall 


Cognitive challenge 


1.0 










Clarity of goals 


0.24 


1.0 








Grading criteria 


0.15 


0.15 


1.0 






Alignment of 
goals/ task 


0.11 


0.11 


0.37* 


1.0 




Alignment of 
goals/grade 


0.16 


0.15 


0.48** 


0.48** 


1.0 


Overall 


0.74*** 


0.37* 


0.37* 


0.32 


0.41* 1.0 


*p < 0.05. **p < 0.01. 


***p < 0.001. 











Table G2 












Interrcorrelation of Ratings for the Writing Assignments (N = 


42) 






Cognitive 

challenge 


Clarity of 
goals 


Grading 

criteria 


Alignment 
goals/ task 


Alignment 

goals /grade Overall 


Cognitive challenge 


1.0 










Clarity of goals 


0.17 


1.0 








Grading criteria 


0.41** 


0.01 


1.0 






Alignment of 
goals/ task 


0.41** 


0.35* 


0.39* 


1.0 




Alignment of 
goals/grade 


0.35* 


0.17 


0.65*** 


0.56*** 


1.0 


Overall 


0.77*** 


0.24 


0.63*** 


0.63*** 


0.52*** 1.0 



*p < 0.05. **p<0.01. ***p< 0.001. 
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Table G3 

Interrcorrelation of Ratings for the Content Writing Assignments (N = 18) 





Cognitive 

challenge 


Clarity of 
goals 


Grading 

criteria 


Alignment 
goals/ task 


Alignment 

goals/grade Overall 


Cognitive challenge 


1.00 










Clarity of goals 


0.34 


1.00 








Grading criteria 


0.28 


-0.23 


1.00 






Alignment of 
goals /task 


0.16 


0.47* 


0.18 


1.00 




Alignment of 
goals /grade 


0.19 


0.37 


0.63** 


0.65** 


1.00 


Overall 


0.55* 


0.64** 


0.11 


0.42 


0.26 1.00 


*p < 0.05. **p < 0.01. 



Table G4 

Interrcorrelation of Ratings for the Challenging Major Project Assignments (N = 50) 




Cognitive 

challenge 


Clarity of 
goals 


Grading 

criteria 


Alignment 
goals/ task 


Alignment 

goals/grade Overall 


Cognitive challenge 


1.00 










Clarity of goals 


0.27 


1.00 








Grading criteria 


0.37* 


0.32* 


1.00 






Alignment of 


0.51*** 


0.46** 


0.52*** 


1.00 




goals/ task 












Alignment of 


0.44** 


0.41** 


079 *** 


0.67*** 


1.00 


goals /grade 












Overall 


0.69*** 


0.52*** 


0.58*** 


0.69*** 


0.60*** 1.00 



*p < 0.05. *y< o.oi. ***p< o.ooi. 




77 



69 



TM033042 




U.S. Department of Education 

Office of Educational Research and Improvement (OERI) 
National Library of Education (NLE) 
Educational Resources Information Center (ERIC) 




NOTICE 

Reproduction Basis 




This document is covered by a signed "Reproduction Release 
(Blanket)" form (on file within the ERIC system), encompassing all 
or classes of documents from its source organization and, therefore, 
does not require a "Specific Document" Release form. 




This document is Federally-funded, or carries its own permission to 
reproduce, or is otherwise in the public domain and, therefore, may 
be reproduced by ERIC without a signed Reproduction Release form 
(either "Specific Document" or "Blanket"). 



EFF-089 (3/2000) 





